Video encoding apparatus and video decoding apparatus

ABSTRACT

While maintaining a high degree of freedom in choosing partition sizes and transformation sizes adapted for local characteristics of videos, the amount of metadata is decreased. A video encoding apparatus ( 10 ) divides an input video into blocks of a prescribed size and encodes the video block by block. The video encoding apparatus is provided with: a prediction parameter determining portion ( 102 ) that decides the block partition structure; a predictive image producing portion ( 103 ) that generates predictive images, partition by partition, as prescribed by the partition structure; a transform coefficient producing portion ( 107 ) which applies one of the frequency transformations included in a prescribed transformation preset to prediction residuals, i.e. the differences between predictive images and the input video; a transform restriction deriving portion ( 104 ) which generates the list of transform candidate, i.e. lists of frequency transformations that can be applied to each partition, on the basis of partition format information; and a variable-length-encoding portion ( 108 ) which, on the basis of the list of transform candidate and the transformation preset, performs variable-length encoding on transformation selection flags.

TECHNICAL FIELD

The present invention relates to a video encoding apparatus thatproduces encoded data by encoding a video, and a video decodingapparatus that reproduces the video from the encoded data of the videothat is transmitted and accumulated.

BACKGROUND ART Introduction and Definitions of Basic Terms

In a block-based video encoding system: an input video to be encoded isdivided into predetermined unit of processing that is referred to as“macro blocks” (hereinafter, “MB”); encoding processing is executed foreach of the MBs; and, thereby, encoded data is produced. When a video isreproduced, encoded data to be decoded is processed for each of the MBs,and a decoded image is produced.

There is a system specified in Non-Patent Literature 1 (H.264/AVC(Advanced Video Encoding)) as a block-based video encoding system thatis widely prevalent at present. According to H.264/AVC, predictiveimages that predict an input video to be divided into MBs are produced,and a prediction residual that is a difference between the input videoand the predictive image is calculated. A transform coefficient isderived by applying a frequency transform as represented by a discretecosine transform (DCT) to the prediction residual. The derived transformcoefficient is variable-length-encoded using a method that is referredto as “CABAL (Context-based Adaptive Binary Arithmetic Encoding)” or“CAVLC (Context-based Adaptive Variable Length Encoding)”. Thepredictive image is produced by intra prediction that uses the spatialcorrelation of the video or inter prediction (motion compensatingprediction) that uses the special correlation of the videos.

<Concept of Partition and Effects Thereof>

According to the inter prediction, an image that approximates an inputvideo of an MB to be encoded is produced by a unit that is referred toas “partition”. One or two motion vector(s) are related to eachpartition. A predictive image is produced by referring to an area thatcorresponds to the MB to be encoded on a local decoded image that isrecorded in a frame memory, based on the motion vector(s). The localdecoded image referred to in this case is called as “reference image”.According to H.264/AVC, such partition sizes are available as “16×16”,“16×8”, “8×16”, “8×8”, “8×4”, “4×8”, and “4×4” in pixels. When a smallpartition size is used, a predictive image can be produced bydesignating each motion vector in fine units and, therefore, thepredictive image can be produced that is close to the input video evenwhen the spatial correlation of the motion is low. On the other hand,when a large partition size is used, the amount of codes can be reducedthat are necessary for encoding a motion vector when the spatialcorrelation of the motion is high.

<Concept of Transform Size and Effects Thereof>

For a prediction residual that is produced using a predictive image,spatial or temporal redundancy of the pixel value of the input video isreduced. In addition, an energy can be concentrated on a low frequencycomponent of a transform coefficient by applying a DCT to the predictionresidual. Therefore, by executing the variable-length-encoding using thebias of the energy, the amount of codes of the encoded data can bereduced compared to that of the case where no predictive image and noDOT are used.

According to H.264/AVC, a system (block-adaptive transform selection) isemployed that selects a DCT adapted to the local property of the videofrom DCTs having plural kinds of transform sizes for the purpose ofincreasing the energy concentration on the low frequency component bythe DCT. For example, when a predictive image is produced using theinter prediction, the DCT can be selected that is applicable to thetransform of the prediction residual, from two kinds of DCTs that are an8×8 DCT and a 4×4 DCT. The 8×8 DCT is effective for a flat area havingrelatively a small amount of high-frequency components because thespatial correlation of the pixel value can be used in a wide range inthe 8×8 DCT. On the other hand, the 4×4 DCT is effective for an areahaving a large amount of high-frequency components such as an area thatincludes a contour of an object. It can be said that, according toH.264/AVC, the 8×8 DCT is the DCT for a large transform size and the 4×4DCT is the DCT for a small transform size.

According to H.264/AVC, the 8×8 DCT and the 4×4 DCT can be selected whenthe area of a partition is equal to or larger than 8×8 pixels. The 4×4DCT can be selected when the area of a partition is smaller than 8×8pixels.

As above, according to H.264/AVC, a suitable partition size and asuitable transform size can be selected corresponding to the degree ofeach of the spatial correlation the pixel value or the spatialcorrelation of the motion vector that are the local properties of avideo. Therefore, the amount of codes of the encoded data can bereduced.

<Description of Adaptive Transform Size Expansion and Partition SizeExpansion>

Recently, high-definition videos have increased that have the resolutionequal to or higher than the “HD (1920 pixels×1080 pixels)”. Compared tothe case of a conventional low-resolution video, in the case of ahigh-definition video, the spatial correlation of the pixel value andthe spatial correlation of the motion vector on a video can take a widerange in a local area in the video. Above all, the high-definition videohas a property that the spatial correlations are high in a local areafor both of the pixel value and the motion vector.

Non-Patent Literature 2 describes a video encoding system according towhich the amount of codes of encoded data is reduced by using theproperty of the spatial correlation in a high-definition video as aboveby expanding the partition size and the transform size in H.264/AVC.

More specifically, partition sizes such as “64×64”, “64×32”, “32×64”,“32×32”, “32×16”, and “16×32” are added in addition to those that arespecified in H.264/AVC. Furthermore, DCT that has three kinds of newtransform sizes of “16×16 DCT”, “16×8 DCT”, and “8×16 DCT” are added inaddition to those that are specified in H.264/AVC.

When the area of a partition is equal to or larger than 16×16 pixels,the 16×16 DCT, the 8×8 DCT, and the 4×4 DCT can be selected. When thepartition size is 16×8, the 16×8 DCT, the 8×8 DOT, and the 4×4 DCT canbe selected. When the partition size is 8×16, the 8×16 DCT, the 8×8 DCT,and the 4×4 DCT can be selected. When the partition size is 8×8, the 8×8DCT, and the 4×4 DCT can be selected. When the area of the partition issmaller than 8×8 pixels, the 4×4 DCT can be selected.

According to the system described in the Non-Patent Literature 2, theamount of codes of the encoded data can be reduced, because thepartition size and the transform size that are adaptive to the localproperty of the video can be selected even for a high-definition videowhich has relatively wide dynamic ranges of spatial correlations of thepixel and the motion vector by switching among the above variouspartition sizes and transform sizes.

PRIOR ART DOCUMENT Non-Patent Literature

-   Non-Patent Literature 1: ITU-T Recommendation H.264 (11/07)-   Non-Patent Literature 2: ITU-T T09-SG16-C-0123

DISCLOSURE OF THE INVENTION Problems that the Invention is to Solve

As above, in the video encoding system, it is effective to increase thekinds of partition sizes and transform sizes that can be selected forreducing the amount of codes of the encoded data. However, a new problemarises that the amount of codes of additional information is increasedthat is required to select the partition size and the transform sizeapplied in decoding in each local area in a video.

According to Non-Patent Literatures 1 and 2, even when the partitionsize is large, a frequency transform whose transform size is small (the4×4 DCT) can be used. However, a large partition tends to be selected inan area having a high spatial correlation of the pixel value and themotion vector. Therefore, when the frequency transform whose transformsize is small is applied to such a partition, it is difficult toconcentrate the energy of the prediction residual on fewer transformcoefficients compared to the case where the frequency transform whosetransform size is large is applied thereto. Therefore, a frequencytransform whose transform size is small is rarely selected, and theadditional information is wasted that is necessary for selecting thetransform size. Especially, when the difference is increased in themagnitude between a large partition size and a small transform size dueto expansion of the largest partition size, it becomes more difficultfor a smaller transform size to be selected.

According to Non-Patent Literature 2, a frequency transform can beselected whose transform size is the same as the size of the partitionfor a rectangular partition. However, Non-Patent Literature 2 does notrefer to anything about what criterion is used to determine thetransform sizes that can be selected when kinds of transform size arefurther added.

The present invention was conceived in view of the above circumstancesand an object thereof is to provide a video encoding apparatus thatenables the amount of codes of additional information to be reducedwhile maintaining the possibility that the partition size and thetransform size that are adaptive to the local property of a video can beselected when various partition sizes and various transform sizes areavailable in the video encoding apparatus. Another object thereof is toprovide a video decoding apparatus that is able to decode encoded dataencoded by the video encoding apparatus.

Means to Solve the Problems

A first technical means according to the present invention is a videoencoding apparatus that divides an input video into predetermined sizeblocks and executes an encoding processing for each block, comprising: aprediction parameter determining portion that determines a partitionstructure of the block; a predictive image producing portion thatproduces a predictive image for each partition specified by thepartition structure; a transform coefficient producing portion thatapplies any one of transforms included in a predetermined transformpreset to a prediction residual that is a difference between thepredictive image and the input video; a transform candidate derivingportion that produces a list of transform candidate that is a list ofapplicable transforms based on partition shape information; a frequencytransform determining portion that determines a transform selection flagindicating a transform to be applied to each partition based on the listof transform candidate; and a variable-length-encoding portion thatvariable-length-codes the transform selection flag based on the list oftransform candidate and the transform preset.

A second technical means is the video encoding apparatus of the firsttechnical means, further comprising a transform restriction derivingportion that produces a prohibited transform list that is a list oftransforms inapplicable to each partition based on the partition shapeinformation, wherein the list of transform candidate is derived based onthe prohibited transform list and the transform preset.

A third technical means is the video encoding apparatus of the first orthe second technical means, wherein the partition shape information is aratio of a longitudinal length to a lateral length of a partition, or amagnitude relation between the longitudinal length and the laterallength of the partition.

A fourth technical means is the video encoding apparatus of the first orthe second technical means, wherein the partition shape information iseach of the smallest values of a longitudinal length and a laterallength of a partition.

A fifth technical means is the video encoding apparatus of the first orthe second technical means, wherein the partition structure is expressedby a layer structure, and specified that each partition is included ineither layer corresponding to a shape of the partition, and thepartition shape information is a layer that the partition belongs to.

A sixth technical means is the video encoding apparatus of the firsttechnical means, wherein the predetermined transform preset includes atleast one or more transforms whose transform size is a laterally longrectangle whose height is one pixel, and when a lateral length of apartition is longer than a longitudinal length thereof, the list oftransform candidate producing portion includes in the list of transformcandidate a transform whose transform size is a laterally long rectanglewhose height is one pixel.

A seventh technical means is the video encoding apparatus of the secondtechnical means, wherein the predetermined transform preset includes atleast one transform whose transform size is a square and at least onetransform whose transform size is a laterally long rectangle or alongitudinally long rectangle, and when a longitudinal length of apartition and a lateral length thereof do not coincides with each other,the transform restriction deriving portion includes at least one squaretransform in the prohibited transform list.

An eighth technical means is the video encoding apparatus of the secondtechnical means, wherein the predetermined transform preset includes atleast one or more transforms whose transform size is a laterally longrectangle and at least one or more transforms whose transform size is alongitudinally long rectangle, and when a lateral length of a partitionis longer than a longitudinal length thereof, the transform restrictionderiving portion includes in the prohibited transform list a transformwhose transform size is a longitudinally long rectangle.

A ninth technical means is the video encoding apparatus of the secondtechnical means, wherein the predetermined transform preset includes atleast two or more transforms whose transform sizes mutually are in ananalogous relationship, and when each of the smallest values of alongitudinal length and a lateral length of a partition is equal to orlarger than a predetermined threshold value, the transform restrictionderiving portion includes in the prohibited transform list a transformwhose transform size is the smallest among those of transforms havingthe transform sizes bearing an analogous relationship with each other.

A tenth technical means is the video encoding apparatus of the firsttechnical means, wherein the predetermined transform preset includes afirst transform and a second transform bearing an analogous relationshipwith the first transform and having smaller transform size than that ofthe first transform, the partition structure is expressed by a layerstructure and specified that each partition is included in either layercorresponding to a shape of the partition, and the transform restrictionderiving portion includes the first transform in the list of transformcandidate and does not include the second transform in the list oftransform candidate when a partition belongs to a predetermined layerthat is not a lowermost layer, and includes the second transform in thelist of transform candidate when the partition belongs to a layer lowerthan the predetermined layer that is not the lowermost layer.

An eleventh technical means is a video decoding apparatus that executesa decoding processing for input encoded data for each block, comprising:a variable-length code decoding portion that decodes a partitionstructure of a block to be processed from the input encoded data; apredictive image producing portion that produces a predictive image foreach partition that is specified by the partition structure; and atransform control deriving portion that derives a transform restrictionand/or a transform candidate of an applicable transform based onpartition shape information, wherein the variable-length-decodingportion decodes a transform selection flag based on the input decodeddata and the transform restriction and/or the transform candidate aswell as decodes a transform coefficient of the block to be processedbased on the transform selection flag, the video decoding apparatusfurther comprises: a prediction residual reconstructing portion thatreconstructs a prediction residual by applying an inverse transform tothe transform coefficient, the inverse transform corresponding to atransform, the transform being specified by the transform selectionflag; and a local decoded image producing portion that outputs decodedimage data based on the predictive image and the prediction residual,the decoded image data corresponding to the block to be processed.

A twelfth technical means is a video decoding apparatus that decodes animage for each block of the image by processing input encoded data,comprising: a variable-length code decoding portion that decodes a rulefor specifying or updating a transform, applicable to a predictionparameter determining a partition structure of a block to be processed,a partition, and a set of partitions; and a predictive image producingportion that produces a predictive image for each partition specified bythe partition structure, wherein the variable-length-decoding portiondecodes a transform selection flag based on the input encoded data andthe rule as well as decodes a transform coefficient of the block to beprocessed based on the transform selection flag, and the video decodingapparatus further comprises: a prediction residual reconstructingportion that reconstructs a prediction residual by applying an inversetransform corresponding to a transform specified by the transformselection flag to the transform coefficient; and a local decoded imageproducing portion that outputs decoded image data corresponding to theblock to be processed based on the predictive image and the predictionresidual.

A thirteenth technical means is the video decoding apparatus of thetwelfth technical means, wherein the rule includes a rule of including aspecific kind of transform in the list of transform candidate for apartition having a specific shape.

A fourteenth technical means is the video decoding apparatus of thetwelfth technical means, wherein the rule includes a rule specifyingthat inclusion of a specific kind of transform is prohibited in the listof transform candidate for a partition having a specific shape.

A fifteenth technical means is the video decoding apparatus of thetwelfth technical means, wherein when a specific kind of transform isincluded in the list of transform candidate for a partition having aspecific shape, the rule includes a rule that specifies that thetransform is replaced with another transform.

A sixteenth technical means is the video decoding apparatus of thetwelfth technical means, wherein the rule includes a complex rule thatis expressed as a combination of basic rules each of which permits,prohibits, or replaces a specific kind of transform for a partitionhaving a specific shape.

A seventeenth technical means is the video decoding apparatus of thesixteenth technical means, wherein the rule includes, as the complexrule, a rule specifying that inclusion of a small-sized transform oftransforms in an analogous relationship is prohibited in the list oftransform candidate for a partition belonging to a layer that is upperthan a specific layer.

An eighteenth technical means is the video decoding apparatus of thesixteenth or the seventeenth technical means, wherein thevariable-length-encoding portion codes a flag indicating whether thecomplex rule is applied as the rule.

A nineteenth technical means is a video encoding apparatus that dividesan input video into predetermined size blocks and executes an encodingprocessing for each block, comprising: a prediction parameterdetermining portion that determines a partition structure of a block; apredictive image producing portion that produces an predictive image foreach partition, the partition being specified by the partitionstructure; a transform coefficient producing portion that applies anyone of frequency transforms included in a predetermined transform presetto a prediction residual that is a difference between the predictiveimage and the input video; a transform control deriving portion thatderives a transform restriction and/or a transform candidate of anapplicable transform for each partition based on partition shapeinformation; a frequency transform determining portion that determines atransform selection flag indicating a transform to be applied based onthe transform restriction and/or the transform candidate; a transformcandidate deriving rule determining portion that determines a rule,specifying or updating a method of deriving the transform restrictionand/or the transform candidate in the transform control derivingportion; and a variable-length-encoding portion thatvariable-length-codes the transform selection flag based on thetransform restriction and/or a list of transform candidate and thetransform preset, further more variable-length-codes the transformcandidate deriving rule for each predetermined unit that is larger thana block.

Effects of the Invention

According to the video encoding apparatus of the present invention, anamount of codes of additional information is able to be reduced whilemaintaining the possibility at a high level that the transform sizesuitable to the local property of a video is able to be selected; andfurther a processing amount of encoding processing is able to be reducedby limiting the transform sizes that can be selected to highly effectiveones when a specific partition size is selected. According to thedecoding apparatus of the present invention, encoded data is able to bedecoded that is encoded by the video encoding apparatus.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for explaining the definitions of an expanded macroblock (MB) and a processing sequence.

FIG. 2 is a block diagram of an embodiment of a video encoding apparatusof the present invention.

FIG. 3 is a diagram for explaining the definitions of a partition layerstructure and a processing sequence.

FIG. 4 is a flowchart for explaining an example of producing processingof a prohibited transform list.

FIG. 5 is a flowchart for explaining another example of the producingprocessing of the prohibited transform list.

FIG. 6 are diagrams for explaining partition division executed when theprohibited transform list is produced.

FIG. 7 are other diagrams for explaining the partition division executedwhen the prohibited transform list is produced.

FIG. 8 is a flowchart for explaining yet another example of theproducing processing of the prohibited transform list.

FIG. 9 is a diagram for explaining a specific example of the productionprocedure of the prohibited transform list.

FIG. 10 is a flowchart for explaining an exemplary encoded dataproducing processing for a transform selection flag.

FIG. 11 is a block diagram of an embodiment of a video decodingapparatus of the present invention.

FIG. 12 is a block diagram of another embodiment of the video encodingapparatus of the present invention.

FIG. 13 is a flowchart for explaining an example of producing processingof a list of transform candidate.

FIG. 14 is a block diagram of another embodiment of the video decodingapparatus of the present invention.

FIG. 15 is a block diagram of yet another embodiment of the videoencoding apparatus of the present invention.

FIG. 16 is a block diagram of yet another embodiment of the videodecoding apparatus of the present invention.

MODES FOR CARRYING OUT THE INVENTION First Embodiment

A video encoding apparatus 10 and a video decoding apparatus 20 that arean embodiment of a video encoding apparatus and a video decodingapparatus according to the present invention will be described withreference to FIGS. 1 to 11. In the description of these drawings, thesame reference numerals are given to the same components and thedescriptions of the same components are omitted.

In the following description, it is assumed that input videos aresequentially input to the video encoding apparatus by expanded MB thatis configured by 64×64 pixels and processing is executed therefor. Aninput order of the expanded MBs is assumed to be the order of a rasterscanning as shown as FIG. 1. However, the present invention isapplicable to the case where the size of the expanded MB is the oneother than those above. Especially, the present invention is effectivefor expanded MBs having a size larger than a size of 16×16 pixels thatis a unit amount to be processed widely used at present.

It is assumed that the processing executed in the video encodingapparatuses and the video decoding apparatuses in the followingdescription is realized based on H.264/AVC, and that portions that donot specifically mention the operations follow the operations accordingto H.264/AVC. However, the video encoding system that the presentinvention is applicable to is not limited to H.264/AVC and the presentinvention is applicable to systems that are similar to H.264/AVC such asVC-1, MPEG-2, and AVS, and other video encoding system that employsprocessing for each block or a frequency transform.

<Configuration of Video Encoding Apparatus 10>

FIG. 2 is a block diagram of the configuration of the video encodingapparatus 10. The video encoding apparatus 10 includes a frame memory101, a predictive parameter determining portion 102, a predictive imageproducing portion 103, a transform restriction deriving portion 104, afrequency transform determining portion 105, a prediction residualproducing portion 106, a transform coefficient producing portion 107, avariable-length-encoding portion 108, a prediction residualreconstructing portion 109, and a local decoded image producing portion110.

<Frame Memory 101>

The frame memory 101 has a local decoded image recorded thereon. The“local decoded image” is an image that is produced by adding apredictive image to a prediction residual that is reconstructed byapplying an inverse frequency transform to a transform coefficient. Atthe time when a specific expanded MB of a specific frame of the inputvideo is processed, a local decoded image for a frame that is encodedprior to a frame to be processed and a local decoded image thatcorresponds to an expanded MB that is encoded prior to the expanded MBto be processed, are recorded in the frame memory 101. It is assumedthat the local decoded image recorded in the frame memory 101 cansuitably be read by each of the components in the apparatus.

<Prediction Parameter Determining Portion 102 (Definition of PartitionStructure, Description of Mode Determination)>

The prediction parameter determining portion 102 determines a predictionparameter based on a local property of the input video and outputs theprediction parameter. The prediction parameter includes at least apartition structure that expresses the structure of partitions that areapplied to each portion in an expanded MB, and motion information forinter prediction (a motion vector and an index of a local decoded imageto be referred to (reference image index)). The prediction parameter mayalso include intra prediction mode that indicates a predictive imageproducing method for intra prediction.

The details of the partition structure will be described with referenceto FIG. 3. The partition structure is expressed by a hierarchicalstructure. A layer in which 64×64 pixels is handled as a unit amount forprocessing is defined as “layer L0”. A layer in which 32×32 pixels ishandled as a unit amount for processing is defined as “layer L1”. Alayer in which 16×16 pixels is handled as a unit amount for processingis defined as “layer L2”. A layer in which 8×8 pixels is handled as aunit amount for processing is defined as “layer L3”. In each layer, anyone type of divisions can be selected as a dividing method therefor,that are one-division that does not execute any division, horizontaltwo-division that divides an area into two equal areas using a straightline in the horizontal direction, vertical two-division that divides anarea into two equal areas using a straight line in the verticaldirection, and four-division that divides an area into four equal areasusing two straight lines in the horizontal direction and the verticaldirection. A layer whose unit of processing is large is referred to as“upper layer” and a layer whose unit of processing is small is referredto as “lower layer”. In the embodiment, the layer L0 is the uppermostlayer and the layer L3 is the lowermost layer. The partition structureis expressed by identifying the dividing method in each layersequentially from the layer L0 that is the uppermost layer. Morespecifically, the partition structure can uniquely be expressedaccording to the following procedure.

(Step S10) When the dividing method for the layer L0 is any one of theone-division, the horizontal two-division, and the verticaltwo-division, an area expressed using the dividing method is determinedto be the partition of the unit of processing in the layer L0. When thedividing method is the four-division, the partition is determinedaccording to step S11 for each of the divided areas.(Step S11) When the dividing method for the layer L1 is any one of theone-division, the horizontal two-division, and the verticaltwo-division, an area expressed using the dividing method is determinedto be a partition of the unit of processing in the layer L1. When thedividing method is the four-division, a partition is determinedaccording to step S12 for each of the divided areas.(Step S12) When the dividing method for the layer L2 is any one of theone-division, the horizontal two-division, and the verticaltwo-division, an area expressed using the dividing method is determinedto be a partition of the unit of processing in the layer L2. When thedividing method is the four-division, a partition is determinedaccording to step S13 for each of the divided areas.(Step S13) An area expressed using the dividing method for the layer L3is determined to be a partition of the unit of processing in the layerL3.

Processing order of each partition in an expanded MB will be described.As depicted in FIG. 3, in each layer, processing is executed in rasterscanning order regardless of the dividing method. However, when thefour-division is selected as the dividing method in a layer other thanthe lowermost layer (the layer L3), the partition expressed for a lowerlayer is processed in raster scanning order for each of areas acquiredby the four-division. In portions that will be described later, theabove processing order will be applied when processing a partition in anexpanded MB.

A layer Lx to which a partition p belongs is derived according to thefollowing procedure.

(Step S20) When the size of the partition p is equal to the size of apartition that is produced by the one-division, the horizontaltwo-division, or the vertical two-division of a specific layer Ly, thevalue of Lx is set to be “Ly”.(Step S21) In the case other than the above, the value of Lx is set tobe “L3” (Lx is set to be the lowermost layer).

<Description of Partition Shape>

Information characterizing each partition that belongs to the partitionstructure, that is, the partition size, information indicating thefeature of the partition size, or the layer in the partition structureis referred to as “partition shape information”. All of such items areeach partition shape information as, for example, the partition sizeitself such as 32×32, information that indicates whether the partitionsize is larger than a specific partition size, the ratio of thelongitudinal length to the lateral length of the partition, themagnitude relation between the longitudinal length and the laterallength of the partition, the minimal value and the maximal value of thelongitudinal length and the lateral length of the partition, and thelayer that the partition belongs to.

The prediction parameter is determined by a rate distortiondetermination. In the rate distortion determination, for each predictionparameter candidate, the amount of codes of the encoded data acquiredwhen the expanded MB to be processed is encoded using the predictionparameter is calculated and a cost referred to as “rate distortion cost”is calculated from the local decoded image and distortion of the inputvideo, and a prediction parameter is selected that minimizes the cost.The rate distortion cost is calculated for all possible combinations ofthe partition structures and pieces of motion information that areprediction parameters. The best combination thereof is determined to bethe prediction parameter. Denoting the amount of codes of the encodeddata of the expanded MB as “R” and a mean squared error between theinput video and the local decoded image that correspond to the expandedMB as “D”, the rate distortion cost C can be calculated according to anequation C=D+λR using a parameter λ that represents the relation betweenthe amount of codes R and the error D.

According to the rate distortion determination, the prediction parameterthat is suitable for encoding the expanded MB to be processed, that is,a suitable partition structure and motion information that correspondsto each partition are determined and output.

When the rate distortion cost is calculated for a specific predictionparameter, the frequency transform may not uniquely be determined thatis applied to the expanded MB to be processed. In this case, the ratedistortion cost can be used as a rate distortion cost that is acquiredwhen a specific frequency transform is applied, or a minimal ratedistortion cost can be used that is acquired when all of a plurality offrequency transforms are applied.

<Predictive Image Producing Portion 103>

The predictive image producing portion 103 produces a predictive imageof the expanded MB to be processed based on the prediction parameterinput thereinto, and outputs the predictive image. The production of thepredictive image is executed according to the following procedure.

(Step S30) Based on the partition structure included in the predictionparameter, the expanded MB is divided into partitions and a predictiveimage for each of the partitions is produced according to step S31.(Step S31) The motion information corresponding to the partition to beprocessed, that is, the motion vector and the reference image index areread from the prediction parameter. On a local decoded image indicatedby the reference image index, the predictive image is produced by motioncompensating prediction based on the pixel value of the area representedby the motion vector.

<Prediction Residual Producing Portion 106>

The prediction residual producing portion 106 produces a predictionresidual of an expanded MB based on the input video and the predictiveimage that are input thereinto and outputs the prediction residual. Aprediction residual is two-dimensional data that has the same size asthat of the expanded MB and each element thereof is a difference valuebetween a pixel of the input video and a corresponding pixel of thepredictive image.

<Transform Coefficient Producing Portion 107>

The transform coefficient producing portion 107 applies the frequencytransform to the prediction residual based on the prediction residualand a transform selection flag that are input thereinto, thereby,produces a transform coefficient, and outputs the transform coefficient.A transform selection flag indicates the frequency transform to beapplied to each partition of an expanded MB. The transform coefficientproducing portion 107 selects the frequency transform indicated by thetransform selection flag for each partition in the expanded MB andapplies the frequency transform selected to the prediction residual. Thefrequency transform indicated by the transform selection flag is any oneof the frequency transforms included in a set (transform preset) of allfrequency transforms that are applicable by the transform coefficientproducing portion 107.

The transform preset in the embodiment includes nine kinds of frequencytransforms that are a 4×4 DCT, an 8×8 DCT, a 16×16 DCT, a 16×8 DCT, an8×16 DCT, a 16×1 DCT, a 1×16 DCT, an 8×1 DCT, and a 1×8 DCT. Each of thefrequency transforms specified herein corresponds to a DCT (DiscreteCosine Transform) having a specific transform size (for example, the 4×4DCT corresponds to a discrete cosine transform whose transform size is4×4 pixels). The present invention is not limited only to set of theabove frequency transforms, and is also adaptable to any subset of thetransform presets. Frequency transforms including discrete cosinetransforms that have other transform sizes such as, for example, a 32×32DCT and a 64×64 DCT, may be included in the transform preset. Frequencytransforms other than the discrete cosine transform, such as, forexample, a Hadamard transform, a sine transform, and a wavelettransform, or frequency transforms including transforms that approximatethese transforms may be included in the transform preset.

Processing that applies a frequency transform having a transform size ofW×H to a partition of M×N pixels is the processing that is indicated bya pseudo code as follows. An area R (x, y, w, h) means an area that ispresent at a position displaced rightward by x pixels and downward by ypixels from the uppermost and leftmost point in a partition as thestarting point and that has a width of w pixels and a height of hpixels.

for (j=0, j<N, j+=H) { for (i=0, i<M, i+=W) { The frequency transform isapplied to an area R (i, j, W, H) } }

<Transform Restriction Deriving Portion 104>

The transform restriction deriving portion 104 derives a restrictionconcerning the frequency transform that can be selected in eachpartition in the expanded MB as a transform restriction based on theprediction parameter that is input thereinto, and outputs the transformrestriction. The transform restriction deriving portion 104 derives thetransform restriction of each partition based on the partition shapeinformation of the partition that is determined by the predictionparameter.

The transform restriction is defined as a set of prohibited transformlists that are correlated with the partitions in the expanded MB. Aprohibited transform list includes as its elements frequency transformsthat can not be selected for their correlated partition (prohibitedfrequency transforms) of the frequency transforms that are included inthe transform preset. In other words, the remainder that is obtained byextracting the elements of the prohibited transform list from theelements of the transform preset constitutes a set of the frequencytransform that can be selected for the correlated partition (list oftransform candidate).

The prohibited transform list and the list of transform candidate can beindicated by transform set information that includes informationindicating whether each transform is included in the set. Denoting thenumber of transforms included in the transform preset as “Nt”, thenumber of combinations of the transforms is the Nt-th power of two and,therefore, transforms included in the set can be expressed by transformset information that has a range of values from zero to 2^(Nt-1) (theNt-th power of two minus one). Not all combinations of the transformsalways need to be expressed by the transform set information and a valuecorresponding to a specific combination may be expressed. In astraightforward example, when the transform preset includes only the 4×4DCT and the 8×8 DCT, a prohibition list can be expressed by a one-bitflag that indicates whether the 4×4 DCT (or the 8×8 DCT) is prohibited.The list of transform candidate can also be expressed by values of zeroto two by respectively relating the 4×4 DCT to zero, a combination ofthe 4×4 DCT and the 8×8 DCT to one, and the 8×8 DCT to two.

The meaning of the transform set information may be changed for each ofthe layer, the partition, a combination of blocks, etc. The same value“zero” of the transform set information may mean the 16×16 DCT for thelayer L0, the 8×8 DCT for the layer L1, and the 4×4 DCT for the layerL2. The prohibited transform list and the list of transform candidatecan each be expressed using values in a small range by changing themeaning of the values of the transform set information.

Therefore, it is assumed that the transform restriction and the list oftransform candidate in the present invention are considered to be equalto the transform set information that indicates the transformrestriction and the transform candidate without being swayed by the term“list”.

A prohibited transform list Lp for a specific partition p is producedaccording to the following procedure. It is assumed that the size of thepartition p is M×N pixels (laterally M pixels and longitudinally Npixels) and that the partition p belongs to the layer Lx.

(Step S40) Lp is set to be empty.(Step S41) Frequency transform is added to Lp, whose transform size islarger than M×N pixels.(Step S42) Frequency transform is added to Lp, that is determinedcorresponding to the value of Min(M, N).(Step S43) Frequency transform is added to Lp, that is determinedcorresponding to the value of M÷N.(Step S44) Frequency transform that is determined depending on the valueof the layer Lx is added to Lp.

Note that partition shape information includes information indicatingwhether the transform size is larger than M×N pixels, the value ofMin(M, N), the value of M±N, and the value of the layer Lx.

<Limitation on Transform Size According to Min(M, N)>

A more detailed procedure of step S42 will be described with referenceto a flowchart of FIG. 4.

(Step S50) When Min(M, N) is equal to or larger than a predeterminedthreshold value Th1 (for example, Th1 is Th1=16 pixels), the procedureis advanced to step S51 and, when Min(M, N) takes another value, theprocedure is advanced to step S52.(Step S51) When there exist two or more frequency transforms that havetransform sizes in an analogous relationship in a frequency transformlist, the frequency transform, whose transform size is the smallest (the4×4 DCT, the 8×8 DCT, or the 1×8 DCT) in a set of the frequencytransforms having the transform sizes in an analogous relationship isadded to Lp, and the procedure is advanced to step S52. The analogousrelationship in this case includes a similarity relationship. Forexample, transform sizes such as 16×16, 8×8, and 4×4 in the transformpreset of the embodiment are in an analogous relationship. The analogousrelationship also includes an approximate analogous relationship. Forexample, the transform sizes of 16×1 and 8×1 and the transform sizes of1×16 and 1×8 in the transform preset of the embodiment are in ananalogous relationship. Though not applied in the following description,frequency transforms can be classified into three categories of asquare, a longitudinally long rectangle, and a laterally long rectanglebased on the sizes thereof and the frequency transforms belonging toeach of the categories can be regarded as being in an analogousrelationship.(Step S52) When Min(M, N) is equal to or larger than a predeterminedthreshold Th2 (for example, Th2 is Th2=32 pixels), the procedure isadvanced to step S53 and, in other cases, the processing comes to anend.(Step S53) When there exist three or more frequency transforms that havetransform sizes in an analogous relationship in the transform preset,the frequency transform, whose transform size is the second smallest(the 8×8 DCT) in each combination of the frequency transforms that havethe transform sizes in the analogous relationship is added to LP, andthe processing comes to an end. However, Th1 and Th2 are Th2>Th1.

A partition is a unit of motion compensation. The partitionconfiguration is determined such that the motions between the frames ofthe image in the partition are uniform in order to bring a predictiveimage that is produced partition by portion using motion vectors closeto the input image. A large partition is allocated to a large object (ora portion thereof) in the input video and a small partition is allocatedto a small object therein. Generally, in the input video, the spatialcorrelation of the pixel values in an area corresponding to a largeobject is high compared to the spatial correlation of the pixel valuesof an area corresponding to a small object. Therefore, a frequencytransform whose transform size is large is effective compared to afrequency transform whose transform size is small for a large partition.Therefore, even when a frequency transform whose transform size issomewhat small is determined to be a prohibited transform for a largepartition, the amount of codes of the decoded data is not substantiallyincreased.

<Limitation on Transform Size According to Value of M÷N>

A more detailed procedure of step S43 will be described with referenceto a flowchart of FIG. 5.

(Step S60) When the value of M÷N is equal to or larger than two (thelateral length of the partition p is two or more times as long as thelongitudinal length thereof), the procedure is advanced to step S61 and,in other cases, the procedure is advanced to step S63.(Step S61) All frequency transforms that have square transform sizes(the 4×4 DCT, the 8×8 DCT, and the 16×16 DCT) are added to Lp, and theprocedure is advanced to step S62.(Step S62) The frequency transforms whose longitudinal lengths of theirtransform sizes are longer than the lateral length thereof (the 8×16 DCTand the 1×16 DCT) are added to Lp, and the procedure comes to an end.(Step S63) When the value of M÷N is equal to or smaller than 0.5 (thelongitudinal length of the partition p is two or more times as long asthe lateral length thereof), the procedure is advanced to step S64 and,in other cases, the procedure is advanced to step S66.(Step S64) All frequency transforms that have square transform sizes(the 4×4 DCT, the 8×8 DCT, and the 16×16 DCT) are added to Lp, and theprocedure is advanced to step S65.(Step S65) Frequency transforms whose lateral lengths of their transformsizes are longer than the longitudinal length thereof (the 16×8 DCT andthe 16×1 DCT) are added to Lp, and the procedure comes to an end.(Step S66) When the value of M÷N is equal to one (the lateral and thelongitudinal lengths of the partition p are equal), the procedure isadvanced to step S67 and, in other cases, the procedure comes to an end.(Step S67) Frequency transforms that have transform sizes having laterallengths and longitudinal lengths that are different from each other (the16×8 DCT, the 16×1 DCT, the 8×16 DCT, and the 1×16 DCT) are added to Lp.

Intention of steps S61 and S62 will be described with reference to FIG.6. As depicted in FIG. 6( a), it is assumed that two objects (aforeground object O and a background B) are present in a unit ofprocessing, U, in a layer and a border between the foreground object Oand the background B is present in the lower portion of the unit ofprocessing, U. In this case, a partition of a laterally long rectanglewhich the value of M÷N is equal to or larger than two as depicted inFIG. 6( b) is selected. In contrast, no partition of a longitudinallylong rectangle as depicted in FIG. 6( c) is selected.

The relation between the transform size and the amount of codes of theencoded data for a partition that includes both of the background B andthe foreground object O in the case where a laterally long rectangularpartition is selected will be described with reference to FIGS. 6( d) to(f). FIGS. 6( d), (e), and (f) depict the relations between thepartition and the transform size in the cases where a square, alaterally long rectangular, and a longitudinally long rectangulartransform sizes are applied to the partition. When the frequencytransform whose transform size is a square (FIG. 6( d)) or the frequencytransform whose transform size is a longitudinally long rectangle (FIG.6( f)) is used, a border tends to be present in the area to which thefrequency transform is applied.

On the other hand, when the frequency transform having a transform sizeof a laterally long rectangle (FIG. 6( e)) is used, a border rarelypresents in the area to which the frequency transform is applied. When aborder is present in the area to which the frequency transform isapplied, the energy can not be concentrated on the low frequencycomponent of the transform coefficient due to the frequency transformand, therefore, the amount of codes required for the encoding of thetransform coefficient is increased. On the other hand, when no border ispresent in the area to which the frequency transform is applied, theenergy can be concentrated on the low frequency component of thetransform coefficient due to the frequency transform and, therefore, theamount of codes required for the encoding of the transform coefficientis reduced. Therefore, for a laterally long rectangular partition, it ismore effective to apply the frequency transform whose transform size isa laterally long rectangle compared to the case where the frequencytransform whose transform size is a square or a longitudinally longrectangle is applied. Therefore, even when the frequency transform whosetransform size is a square or a longitudinally long rectangle is set tobe a prohibited transform for a laterally long rectangular partition,the amount of codes of the encoded data is not substantially increased.

Intention of each of steps S64 and S65 is the same as above. Even whenthe frequency transform whose transform size is a square or a laterallylong rectangle is set to be a prohibited transform for a longitudinallylong rectangular partition, the amount of codes of the encoded data isnot substantially increased.

Intention of step S66 will be described with reference to FIG. 7. Asdepicted in FIG. 7( a), it is assumed that two objects (the foregroundobject O and the background B) are present in the unit of processing, U,in a layer and the border between the foreground object O and thebackground B is present in the lower-right portion of the unit ofprocessing, U. In this case, a partition is selected that is a square toacquire the value of M÷N to be one as depicted in FIG. 7( b).

The relation between the transform size and the amount of codes of theencoded data for a partition (the lower-right partition) that includesboth of the background B and the foreground object O, acquired when asquare partition is selected will be described with reference to FIGS.7( d) to (f). FIGS. 7( d), (e), and (f) depict the relations between thepartition and the transform sizes for the cases where a square, alaterally long rectangular, and a longitudinally long rectangulartransform sizes are applied to the lower-right partition. In this case,when any one of the transform sizes of a square, a longitudinally longrectangle, and a laterally long rectangle is used the rate of thepresence of a border does not vary so much in the area to which thefrequency transform is applied. Therefore, for the lower-rightpartition, the difference in the amount of codes of the encoded data issmall when the frequency transform is used whose transform size is anyone of the square, the longitudinally long rectangle, and the laterallylong rectangle.

On the other hand, only the background B is included and no border ispresent in a partition other than the lower-right partition in the unitof processing, U. Therefore, when any one of the transform sizes isused, no border is present in the area to which the frequency transformis applied. Therefore, more energy can be concentrated on the transformcoefficient when the frequency transform is used whose transform size isa square with which the spatial relation of the pixel values of theprediction residuals can be utilized in a balanced manner in bothdirections, the horizontal direction (lateral direction) and thevertical direction (longitudinal direction), compared to the case wherethe frequency transform whose transform size is a longitudinally longrectangle or a laterally long rectangle is used. Therefore, for a squarepartition, the frequency transform whose transform size is a square ismore effective than the frequency transform whose transform size is alaterally long rectangle or a longitudinally long rectangle. Therefore,even when the frequency transform whose transform size is a laterallylong rectangle or a longitudinally long rectangle is set to be aprohibited transform for a square partition, the amount of codes of theencoded data is not substantially increased.

<Limitation on Transform Size According to Layer to which PartitionBelongs>

A more detailed procedure of step S44 will be described with referenceto a flowchart of FIG. 8.

(Step S70) When the layer Lx is the uppermost layer, the procedure isadvanced to step S71 and, in other cases, the procedure is advanced tostep S72.(Step S71) Frequency transforms other than the frequency transformhaving the largest transform size (the 8×8 DCT and the 4×4 DCT) of aplurality of candidate frequency transforms having transform sizes whoseshapes (the 16×16 DCT, the 8×8 DOT, and the 4×4 DCT) are added to Lp andthe procedure comes to an end.(Step S72) When the layer Lx is the lowermost layer, the procedure isadvanced to step S73 and, in other cases, the procedure comes to an end.(Step S73) Frequency transforms other than the frequency transformhaving the smallest transform size (the 16×16 DCT and the 8×8 DCT) ofthe plurality of candidate frequency transforms having transform sizeswhose shapes (the 16×16 DCT, the 8×8 DCT, and the 4×4 DCT) are added toLp and the procedure comes to an end.

In the case where partitions are expressed by a layer structure, evenwhen some frequency transforms whose transform sizes are relativelysmall are restricted for a partition that belongs to the uppermostlayer, the amount of codes of the encoded data is not substantiallyincreased. This is because, even when a specific transform (for example,the 8×8 DCT or the 4×4 DCT) can not be selected in the uppermost layer,this transform can be selected in a lower layer. In an area wherefrequency transforms of small transform sizes are effective, nopartition is selected that belongs to the uppermost layer and partitionsare selected that are in lower layers and for which frequency transformsof small transform sizes are can be selected and, thereby, any increaseof the amount of codes of the encoded data can be suppressed.Especially, based on the fact that the frequency transforms of largetransform sizes are effective for a large partition, when a plurality offrequency transforms whose transform sizes are similar in shape arepresent among the candidate frequency transforms, it is preferable torestrict, frequency transforms of small transform sizes among thosefrequency transforms in the uppermost layer.

Similarly, in the case where partitions are expressed by a layerstructure, even when some frequency transforms whose transform sizes arerelatively large are restricted for a partition that belongs to thelowermost layer, the amount of codes of the encoded data is notsubstantially increased. Especially, based on the fact that thefrequency transforms whose transform sizes are small are effective for asmall partition, when a plurality of frequency transforms whosetransform sizes are similar in shape are present among the candidatefrequency transforms, it is preferable to restrict, frequency transformsof large transform sizes among those frequency transforms in thelowermost layer.

<Specific Example of Prohibited Transform List Producing Processing>

A specific example of a procedure for producing transform restrictionsfor a specific partition structure, that is, a prohibited transform listfor each partition executed by the transform restriction derivingportion 104 will be introduced with reference to FIG. 9. As depicted inFIG. 9, an expanded MB is divided into four in the layer L0 and,thereafter, the upper-left portion thereof is divided into one (apartition “a”) in the layer L1, the upper-right portion thereof ishorizontally divided into two (partitions b and c) in the layer L1, thelower-left portion thereof is vertically divided into two (partitions dand e) in the layer L1, and the lower-right portion thereof is dividedinto four in the layer L1.

As to an area that is divided into four in the layer L1, the upper-leftportion thereof is divided into one (partition f) in the layer L2, theupper-right portion thereof is horizontally divided into two (partitionsg and h) in the layer L2, the lower-left portion thereof is verticallydivided into two (partitions i and j) in the layer L2, and thelower-right portion thereof is divided into four in the layer L2. Eachportion acquired by the four-division in the layer L2 is divided intoone (partitions k, l, m, and n) in the layer L3. The transform sizes ofthe frequency transforms that can be selected have, as above, nine kindsof sizes that are 4×4, 8×8, 16×16, 16×1, 1×16, 8×1, 1×8, 16×8, and 8×16.

The partition “a” has a size of 32×32 pixels and belongs to the layerL1. Applying thereto the above procedure for producing the prohibitedtransform list, frequency transforms: whose transform sizes are 4×4,8×1, and 1×8 are added to the prohibited transform list at step S51,frequency transform whose transform size is 8×8 is added to theprohibited transform list at step S52, and frequency transforms whosetransform sizes are 1×16, 16×1, 16×8, and 8×16 are added to theprohibited transform list at step S67.

The partitions b and c have a size of 32×16 pixels and belong to thelayer L1. Applying thereto the above procedure for producing theprohibited transform list, frequency transforms: whose transform sizesare 4×4, 8×1, and 1×8 are added to the prohibited transform list at stepS51, frequency transforms whose transform sizes are 4×4, 8×8, and 16×16are added to the prohibited transform list at step S61, and frequencytransforms whose transform sizes are 1×16 and 8×16 are added to theprohibited transform list at step S62.

The partitions d and e have a size of 16×32 pixels and belong to thelayer L1. Applying thereto the above procedure for producing theprohibited transform list, frequency transforms: whose transform sizesare 4×4, 8×1, and 1×8 are added to the prohibited transform list at stepS51, frequency transforms whose transform sizes are 4×4, 8×8, and 16×16are added to the prohibited transform list at step S64, and frequencytransforms whose transform sizes are 16×1 and 16×8 are added to theprohibited transform list at step S65.

The partition f has a size of 16×16 pixels and belongs to the layer L2.Applying thereto the above procedure for producing the prohibitedtransform list, frequency transforms: whose transform sizes are 4×4,8×1, and 1×8 are added to the prohibited transform list at step S51, andfrequency transforms whose transform sizes are 16×1, 1×16, 16×8, and8×16 are added to the prohibited transform list at step S67.

The partitions g and h have a size of 16×8 pixels and belong to thelayer L2. Applying thereto the above procedure for producing theprohibited transform list, frequency transforms: whose transform sizesare 16×16, 1×16, and 8×16 are added to the prohibited transform list atstep S41, frequency transforms whose transform sizes are 4×4, 8×8, and16×16 are added to the prohibited transform list at step S61, andfrequency transforms whose transform sizes are 1×16 and 8×16 are addedto the prohibited transform list at step S62.

The partitions i and j have a size of 8×16 pixels and belong to thelayer L2. Applying thereto the above procedure for producing theprohibited transform list, frequency transforms: whose transform sizesare 16×16, 16×1, and 16×8 are added to the prohibited transform list atstep S41, frequency transforms whose transform sizes are 4×4, 8×8, and16×16 are added to the prohibited transform list at step S64, andfrequency transforms whose transform sizes are 16×1 and 16×8 are addedto the prohibited transform list at step S65.

The partitions k, l, m, and n have a size of 8×8 pixels and belong tothe layer L3. Applying thereto the above procedure for producing theprohibited transform list, frequency transforms: whose transform sizesare 16×16, 16×1, 16×8, 1×16, and 8×16 are added to the prohibitedtransform list at step S41, frequency transforms whose transform sizesare 16×1, 16×8, 1×16, and 8×16 are added to the prohibited transformlist at step S67, and frequency transforms whose transform sizes are 8×8and 16×16 are added to the prohibited transform list at step 5 b.

As in the above example, for an expanded MB that has another partitionstructure, a prohibited transform list can also be produced for eachpartition in the expanded MB and can be output as a transformrestriction.

In the above, it is described that all of steps S42, S43, and S44 areexecuted in the procedure for producing the prohibited transform list.However, only some of these may be executed. In the detailed procedureof step S42, only either the determination at step S50 or thedetermination at step S51 may be executed. In the detailed procedure ofstep S43, concerning the determinations, only some of the determinationsexecuted at steps S60, S63, and S66 may be executed or, concerning theprocessing executed after each of the determinations, only either stepS61 or step S62, and either step S64 or step S65 may be executed. In thedetailed procedure of step S44, only either the determination at stepS70 on the determination at step S72 may be executed. When suchsimplification of the procedure is executed, the calculation processingnecessary for producing the prohibited transform list can be reduced.

<Frequency Transform Determining Portion 105>

The frequency transform determining portion 105 determines the frequencytransform to be applied to each partition in the expanded MB using thetransform restriction input thereinto; and outputs the informationthereof as a transform selection flag. A procedure is for determiningthe frequency transform to be applied to the specific partition p asfollows:

(Step S120) The prohibited transform list Lp corresponding to thepartition p is extracted from the transform restriction.(Step S121) A list of transform candidate, Cp, is acquired by taking adifference set between the transform preset and the prohibited transformlist Lp.(Step S122) When the list of transform candidate, Cp, is an empty set,the frequency transform is added to the list of transform candidate, Cp,whose transform size is the smallest of the frequency transforms whosetransform sizes are squares that are included in the transform preset.This step is necessary to avoid the case where no applicable frequencytransform is present when the prohibited transform list coincides withthe transform preset. When the prohibited transform list is alwaysproduced that does not coincide with the transform preset, this step maybe omitted.(Step S123) The rate distortion cost for a case where each of thefrequency transforms included in the list of transform candidate, Cp, isapplied is calculated, and the frequency transform that minimizes therate distortion cost is determined to be the frequency transform to beapplied to the partition p.

<Variable-Length-Encoding Portion 108>

The variable-length-encoding portion 108 produces the encoded data thatcorresponds to the transform coefficient, the prediction parameter, andthe transform selection flag in the expanded MB, based on the transformcoefficient, the prediction parameter, the transform restriction, andthe transform selection flag that are input thereinto, and outputs theencoded data.

The transform coefficient and the prediction parameter arevariable-length-encoded in a conventional method and the resultant codesare output. The transform selection flag is variable-length-encodedusing the transform restriction and the resultant code is output. Aprocedure for variable-length-encoding the transform selection flag willbe described with reference to a flowchart of FIG. 10.

(Step S80) When the dividing method used for the layer L0 in theexpanded MB is other than quartering, the process of step S81 isexecuted and, in other cases, the processes of steps S82 to S92 areexecuted.(Step S81) The information is variable-length-encoded that indicates thefrequency transform to be applied to each partition in the unit ofprocessing of the layer L0 (64×64 pixels), and the procedure comes to anend.(Step S82) Processes of following steps S83 to S92 are executed for eachof unit of processing (32×32 pixels each) in the layer L1 that areacquired by dividing the unit of processing in the layer L0 into four.(Step S83) When the dividing method used for the layer L1 in the currentunit of processing (32×32 pixels) is other than quartering, theprocedure is advanced to step S84 and, in other cases, the procedure isadvanced to step S85.(Step S84) The information is variable-length-encoded that indicates thefrequency transform to be applied to each partition in the current unitof processing (32×32 pixels), and the procedure is advanced to step S92.(Step S85) Processes of following steps S86 to S91 are applied to eachof unit of processing (16×16 pixels each) in the layer L2 that areacquired by dividing the unit of processing in the layer L1 (32×32pixels) into four.(Step S86) When the dividing method used in the layer L2 in the currentunit of processing (16×16 pixels) is other than quartering, theprocedure is advanced to step S87 and, in other cases, the procedure isadvanced to step S88.(Step S87) The information is variable-length-encoded that indicates thefrequency transform to be applied to each partition in the current unitof processing (16×16 pixels), and the procedure is advanced to step S91.(Step S88) Processes of following steps S89 to S90 are executed for eachof unit of processing (8×8 pixels each) in the layer L3 that areacquired by dividing the unit of processing in the layer L2 into four.(Step S89) The information is variable-length-encoded that indicates thefrequency transform to be applied to each partition in the current unitof processing (8×8 pixels), and the procedure is advanced to step S90.(Step S90) When processing of all unit of processing (8×8 pixels each)comes to an end, the procedure is advanced to step S91. When theprocessing does not come to an end, a next unit of processing (8×8pixels) is set and the procedure is advanced to step S89.(Step S91) When processing of all unit of processing (16×16 pixels each)comes to an end, the procedure is advanced to step S92. When theprocessing does not come to an end, a next unit of processing (16×16pixels) is set and the procedure is advanced to step S86.(Step S92) When processing of all unit of processing (32×32 pixels each)comes to an end, the procedure comes to an end. When the processing doesnot come to an end, a next unit of processing (32×32 pixels) is set andthe procedure is advanced to step S83.

The variable-length-encoding is executed according to the followingprocedure for the transform selection flag that corresponds to thespecific partition p.

(Step S130) The prohibited transform list Lp corresponding to thepartition p is extracted from the transform restriction.(Step S131) The list of transform candidate, Cp, is acquired by taking adifference set between the transform preset and the prohibited transformlist Lp.(Step S132) When the list of transform candidate, Cp, is an empty set,the frequency transform is added to the list of transform candidate, Cp,whose transform size is the smallest of the frequency transforms whosetransform sizes are squares that are included in the transform preset.The frequency transform added in this step is not limited to the abovefrequency transforms, and may be the frequency transform whose transformsize is smaller than that of another partition p included in thetransform preset. However, this frequency transform needs to be the samefrequency transform as that used at step S122 of the frequency transformdetermining portion.(Step S133) When the number of frequency transforms included in the listof transform candidate, Cp, is only one, the variable-length-encodingprocessing comes to an end. In this case, even when the informationindicating the frequency transform to be applied to the partition p isnot included in the encoded data, no problem arises because it can beuniquely indentified at the time of decoding the data which frequencytransform must be applied.(Step S134) The frequency transforms included in the list of transformcandidate, Cp, are rearranged in predetermined order and correlated tothe indexes that increase one by one starting from zero.(Step S135) The index variable-length-encoded that is correlated withthe frequency transform to be applied to the partition p. As a method ofvariable-length-encoding of the index, for example, a method that takesa bit string of an index value acquired by expressing the index value inthe binary using t bits, as the encoded data using the minimal “t” withwhich the t-th power of two is equal to or larger than s where s is thenumber of elements of the candidate frequency transform list.

When the number of elements of the candidate frequency transform listbecomes smaller, the amount of codes necessary for encoding the indexbecomes smaller. By setting a prohibited transform for each partition,the amount of codes necessary for encoding the transform selection flagcan be reduced. When the number of elements of the candidate frequencytransform list is small, the amount of computing can be reduced for theencoding processing to select the frequency transform to be applied.

For the predetermined order at step S134, order can be used according towhich, for example, an index smaller than an index attached to afrequency transform whose transform size is small is attached to afrequency transform whose transform size is large, when the transformsize of a frequency transform is a square, an index smaller than anindex attached to another frequency transform whose transform size is alaterally long rectangle is attached to the frequency transform, andwhen the transform size of a frequency transform is a laterally longrectangle, an index smaller than an index attached to another frequencytransform whose transform size is a longitudinally long rectangle isattached to the frequency transform. In this case, indexes in ascendingorder tend to be correlated one by one with the 16×16 DCT, the 16×8 DCT,the 8×16 DCT, the 8×8 DCT, the 4×4 DCT, the 16×1 DCT, the 1×16 DCT, the8×1 DCT, and the 1×8 DCT in this order.

The predetermined order at step S134 may also be descending order of thefrequency of selecting each frequency transform, as another example.More specifically, the number of times that each transform in thetransform preset is selected as the transform of a partition afterstarting the encoding processing for the input video is counted, andorder is produced such that a smaller index is allocated to a frequencytransform that is selected for more times. In this case, a bias isgenerated also in the frequency of production of the index and,therefore, the amount of codes is reduced that is acquired when theindex is variable-length-encoded at step S135. The value of thecoefficient of the number of times of the selection may be initializedto a predetermined value such as zero at a proper timing such as thestarting time point of encoding of a new frame or the starting timepoint of encoding a slice that is a set of a predetermined number ofexpanded MBs. The number of times of selecting a conditional frequencytransform such as, for example, the number of times of selecting eachfrequency transform for each partition size may be counted and used.

Another method may also be used for the variable-length-encoding of theindex executed at step S135. For example, various kinds of VLCs, CABACs,etc., specified in H.264/AVC may also be used.

Without variable-length-encoding the index as it is, a flag is encodedthat indicates whether the index coincides with an index estimated valueand, only when the flag indicates no coincidence, the index may bevariable-length-encoded. The frequency transform used for the partitionto be processed is estimated using the pieces of information on theexpanded MB already encoded (such as the local decoded image, thepartition structure, and the motion vector), and an index correspondingto the frequency transform may be determined to be the index estimatedvalue. Especially, it is preferable to derive an index estimated valuebased on the frequency transform that is applied to a partition in thevicinity of the partition to be processed taking into consideration thespatial correlation of the frequency transform. More specifically, theindexes of the frequency transforms that are applied to the partitionslocated on the left of, above, and on the upper-right of the partitionto be processed are derived, respectively. And a system in which two ormore of those indexes coincide with each other, the value of the two ormore indexes is determined to be the index estimated value, and in othercases, the smallest value of those indexes is determined to be the indexestimated value is preferable.

It is described that a variable-length-encoding is given to thetransform selection flags for all the partitions in the procedure forvariable-length-encoding of the transform selection flag. However, afterimposing a restriction that the frequency transform is commonly appliedto the partitions belonging to the same unit of processing of thespecific layer Lx, one transform selection flag common to the partitionsin the unit of processing may be variable-length-encoded for each unitof processing of the layer Lx. In this case, the degree of freedom ofselecting the frequency transform is lowered. However, encoding of anytransform selection flag is not necessary for each partition and thetransform selection flag only has to be encoded for each unit ofprocessing of the layer Lx and, therefore, the amount of codes can bereduced that are necessary for encoding the transform selection flag. Incontrast, a partition may be divided into units that each are notsmaller than the frequency transform whose transform size is the largestincluded in the list of transform candidate, and the transform selectionflag may be encoded for this unit.

<Prediction Residual Reconstructing Portion 109>

The prediction residual reconstructing portion 109 reconstructs theprediction residual by applying an inverse frequency transform to thetransform coefficient based on the transform coefficient and thetransform selection flag that are input thereinto, and outputs thereconstructed prediction residual. When the transform coefficient isquantized, inverse quantization is applied to the transform coefficientprior to the application of the inverse frequency transform.

<Local Decoded Image Producing Portion 110>

The local decoded image producing portion 110 produces a local decodedimage based on the predictive image and the prediction residual that areinput thereinto, and outputs the local decoded image. Each pixel valueof the local decoded image is the sum of the pixel values of thecorresponding pixels of the predictive image and the predictionresidual. A filter may be applied to the local decoded image with thepurpose of reducing the block distortion that is generated on a blockborder and reducing the quantization errors.

<Operations of Video Encoding Apparatus 10>

Operations of the video encoding apparatus 10 will be described.

(Step S100) The input video externally input into the video encodingapparatus 10 is sequentially input in expanded MBs into the predictionparameter determining portion 102 and the prediction residual producingportion 106. Processes of S101 to S109 as follows are sequentiallyexecuted for each of the expanded MBs.(Step S101) The prediction parameter determining portion 102 determinesa prediction parameter for the expanded MB to be processed based on theinput video that is input thereinto, and outputs the predictionparameter to the predictive image producing portion 103 and thevariable-length-encoding portion 108.(Step S102) The predictive image producing portion 103 produces thepredictive image that approximates an area of the expanded MB to beprocessed in the input video based on the prediction parameter inputthereinto and the local decoded image recorded in the frame memory 101,and outputs the predictive image to the prediction residual producingportion 106 and the local decoded image producing portion 110.(Step S103) The prediction residual producing portion 106 produces theprediction residual that corresponds to the expanded MB to be processedbased on the input video and the predictive image that are inputthereinto, and outputs the prediction residual to the frequencytransform determining portion 105 and the transform coefficientproducing portion 107.(Step S104) The transform restriction deriving portion 104 derives arestriction on the frequency transform in each partition of the expandedMB to be processed as a transform restriction based on the predictionparameter that is input thereinto, and outputs the transform restrictionto the frequency transform determining portion 105 and thevariable-length-encoding portion 108.(Step S105) The frequency transform determining portion 105 determinesthe frequency transform to be applied to each partition of the expandedMB to be processed based on the transform restriction and the predictionresidual that are input thereinto, and outputs the frequency transformas a transform selection flag to the transform coefficient producingportion 107, the variable-length-encoding portion 108, and theprediction residual reconstructing portion 109.(Step S106) The transform coefficient producing portion 107 applies thefrequency transform specified by the transform selection flag inputthereinto to the prediction residual input thereinto, thereby, producesthe transform coefficient that corresponds to the expanded MB to beprocessed, and outputs the transform coefficient to thevariable-length-encoding portion 108 and the prediction residualreconstructing portion 109.(Step S107) The prediction residual reconstructing portion 109 appliesthe inverse frequency transform that corresponds to the frequencytransform specified by the transform selection flag input thereinto tothe transform coefficient input thereinto, thereby, reconstructs theprediction residual that corresponds to the expanded MB to be processed,and outputs the reconstructed prediction residual to the local decodedimage producing portion 110.(Step S108) The local decoded image producing portion 110 produces thelocal decoded image based on the prediction residual and the predictiveimage that are input thereinto, and outputs the local decoded image tothe frame memory 101 to record the local decoded image thereon.(Step S109) The variable-length-encoding portion 108variable-length-codes the transform coefficient, the predictionparameter, and the transform selection flag that are input thereintousing the transform restriction input thereinto, and externally outputsthe resultant data as the encoded data.

According to the above procedure, the video encoding apparatus 10 cancode the input video input thereinto, thereby, produce the encoded data,and externally output the encoded data.

<Configuration of Video Decoding Apparatus 20>

The video decoding apparatus 20 will be described that decodes theencoded data that is encoded by the video encoding apparatus 10 andthat, thereby, produces the decoded video.

FIG. 11 is a block diagram of the configuration of the image decodingapparatus 20. The video decoding apparatus 20 includes the frame memory101, the predictive image producing portion 103, the transformrestriction deriving portion 104, the prediction residual reconstructingportion 109, the local decoded image producing portion 110, and avariable-length code decoding portion 201.

The variable-length code decoding portion 201 decodes the predictionparameter, the transform selection flag, and the transform coefficientbased on the encoded data and the transform restriction that are inputthereinto, and outputs the decoded results. More specifically, theprediction parameter is first decoded from the encoded data and theresult is output. The transform selection flag is then decoded from theencoded data using the transform restriction and the result is output.The transform coefficient is finally decoded from the encoded data usingthe transform selection flag and the result is output.

<Operations of Video Decoding Apparatus 20>

Operations of the video decoding apparatus 20 will be described.

(Step S110) The encoded data externally input into the video decodingapparatus 20 is sequentially input into the variable-length codedecoding portion 201 in expanded MBs, and processes of S111 to S117 asfollows are sequentially executed for the encoded data that correspondsto each expanded MB.(Step S111) The variable-length code decoding portion 201 decodes theprediction parameter that corresponds to the expanded MB to be processedfrom the encoded data that is input thereinto, and outputs theprediction parameter to the predictive image producing portion 103 andthe transform restriction deriving portion 104.(Step S112) The transform restriction deriving portion 104 derives arestriction concerning the frequency transform for each partition of theexpanded MB to be processed as the transform restriction based on theprediction parameter that is input thereinto, and outputs the transformrestriction to the variable-length code decoding portion 201.(Step S113) The variable-length code decoding portion 201 decodes thetransform selection flag that corresponds to the MB to be processedbased on the encoded data and the transform restriction that are inputthereinto, and outputs the transform selection flag to the predictionresidual reconstructing portion 109.(Step S114) The variable-length code decoding portion 201 decodes thetransform coefficient that corresponds to the expanded MB to beprocessed, based on the decoded data input thereinto and the transformselection flag derived at (step S113), and outputs the transformcoefficient to the prediction residual reconstructing portion 109.(Step S115) The predictive image producing portion 103 produces thepredictive image that corresponds to the expanded MB to be processed,based on the prediction parameter input thereinto and the local decodedimage recorded in the frame memory 101, and outputs the predictive imageto the local decoded image producing portion 110.(Step S116) The prediction residual reconstructing portion 109 appliesthe inverse frequency transform corresponding to the frequency transformspecified by the transform selection flag input thereinto, to thetransform coefficient input thereinto, thereby, reconstructs theprediction residual that corresponds to the expanded MB to be processed,and outputs the prediction residual to the local decoded image producingportion 110.(Step S117) The local decoded image producing portion 110 produces thelocal decoded image based on the prediction residual and the predictiveimage that are input thereinto, outputs the local decoded image to theframe memory 101 to record the local decoded image in the memory 101,and externally outputs the local decoded image as the area on thedecoded video that corresponds to the block to be processed.

As above, according to the video decoding apparatus 20, the decodedvideo can be produced from the encoded data that is produced by thevideo encoding apparatus 10.

<Appended Item 1: Use of Information on Items Other than Partition Sizeand Layers to Belong to>

In the description of the video encoding apparatus 10 and the videodecoding apparatus 20, the prohibited transform list for each partitionin the expanded MB is described to be produced based only on thepartition size and the layer to which the partition belongs. However,another piece of information may also be used that can be reproduced forthe decoding based on the information included in the encoded data. Forexample, the motion vector and the reference image index included in theprediction parameter may also be used to derive the prohibited transformlist.

A procedure will be described for adding a frequency transform to theprohibited transform list using a motion vector and a reference imageindex in the specific partition. A motion vector of the partition p isdenoted by “mvp” and a reference image index thereof is denoted by“refp”. A motion vector of a partition (partition u) located at theleftmost position of the partitions that are adjacent to the top side ofthe partition p is denoted by “mvu” and a reference image index thereofis denoted by “refu”. A motion vector of a partition (partition l)located at the top end of the partitions that are adjacent to the leftside of the partition p is denoted by “mvl” and a reference image indexthereof is denoted by “refl”.

(Step S140) When all of mvp, mvu, and mvl coincide with each other andall of refp, refu, and refl coincide with each other, the procedure isadvanced to step S141. In other cases, the procedure comes to an end.(Step S141) When two or more frequency transforms that have transformsizes in an analogous relationship in the frequency transform list arepresent, the frequency transform, whose transform size is the smallestin each combination of the frequency transforms that have the transformsizes in the analogous relationship is added to Lp, and the procedurecomes to an end.

The coincidence of the motion vectors among adjacent blocks means thatthe spatial correlation of the motion vectors is high in the local areain the vicinity of the expanded MB to be encoded. When the spatialcorrelation of the motion vectors is high, the spatial correlation ofthe pixel values tends to be also high and, therefore, the increase ofthe amount of codes of the encoded data is slight even when applicationis prohibited of the frequency transforms whose transform sizes aresmall of the frequency transforms that have similar transform sizes.

In the above, it is assumed that the motion vectors and the referenceimage indexes to be used to derive the prohibited transform list are themotion vectors and the reference image indexes of the partitions thatare adjacent to the partition p. However, other motion vectors may beused. For example, motion vectors may be used in expanded MBs adjacentto the expanded MB to which the partition p belongs (the expanded MB tobe processed). More specifically, a motion vector of a partition locatedon the upper-right in the expanded MB adjacent to the left side of theexpanded MB to be processed is used as mvl, and a motion vector of apartition located on the lower-left in the expanded MB adjacent to thetop side of the expanded MB to be processed is used as mvu. In thiscase, the same mvl and mvu are used in all the partitions in theexpanded MB and, therefore, the processes of steps S140 and S141 can beexecuted in parallel for each partition.

<Appended Item 2: Timing for Producing Prohibited Transform List>

In the description of the video encoding apparatus 10 and the videodecoding apparatus 20, it is described that the transform restrictionderiving portion 104 executes the processing of producing the prohibitedtransform list for each partition of the expanded MB at any time.However, when the addition of the frequency transform to the prohibitedtransform list is executed based only on the partition size and thelayer to which the partition belongs, a prohibited frequency transformlist may also be produced in advance at a predetermined timing. In thiscase, the prohibited transform list produced in advance for each kind ofpartition needs to be correlated with each partition in the expanded MBby the transform restriction deriving portion 104. The predeterminedtiming may be the starting time point of the encoding of the input videoor a time point immediately after the start of the decoding of theencoded data, or may be a time point immediately after the start of theencoding or the decoding processing of a predetermined encoding unitsuch as a sequence, a frame, or a slice. The number of times ofexecuting the processing of producing the prohibited transform list canbe reduced and, therefore, the amount of processing of encoding anddecoding can be reduced.

In contrast, in the case where the frequency transform is added to theprohibited transform list, when the motion vectors and the referenceimage indexes are used, the processing of producing the prohibitedtransform list needs to be executed at any time for each expanded MB asdescribed for the video encoding apparatus 10 and the video decodingapparatus 20. In this case, the amount of processing for the encodingand the decoding is increased due to the increase of the number of timesof executing the processing of producing the prohibited transform list.However, compared to the case where the producing processing is notexecuted for each MB, the prohibited transform list can be produced thatis more adaptive to the local property of the video by using moreinformation than can be derived from the encoded data.

Second Embodiment

A video encoding apparatus 11 and a video decoding apparatus 21 that areanother embodiment of the video encoding apparatus and the videodecoding apparatus according to the present invention will be describedwith reference to FIGS. 12 to 14. In the description of the accompanyingdrawings, the same components are given the same reference numerals andwill not again be described.

The video encoding apparatus 11 and the video decoding apparatus 21 inthe embodiment are characterized in that the list of transform candidateis directly derived without producing any prohibited transform list byreplacing the transform restriction deriving portion 104 in each of thevideo encoding apparatus 10 and the video decoding apparatus 20 with atransform candidate deriving portion 111.

The transform restriction deriving portion 104 and the transformcandidate deriving portion 111 are collectively referred to as“transform control deriving portion”.

FIG. 12 is a block diagram of the configuration of the video encodingapparatus 11. The video encoding apparatus 11 includes the frame memory101, the prediction parameter determining portion 102, the predictiveimage producing portion 103, the prediction residual producing portion106, the transform coefficient producing portion 107, the predictionresidual reconstructing portion 109, the local decoded image producingportion 110, the transform candidate deriving portion 111, a frequencytransform determining portion 112, and a variable-length-encodingportion 113.

The transform candidate deriving portion 111 outputs, as a list oftransform candidate, information on frequency transforms that can beselected for each partition in an expanded MB based on a predictionparameter that is input thereinto. The transform candidate derivingportion 111 produces a list of transform candidate for the partitionbased on partition shape information of each partition determined by theprediction parameter.

The list of transform candidate is correlated with each partition in theexpanded MB and specifies a set of frequency transforms that can beselected for each partition of frequency transforms included in atransform preset.

A list of transform candidate, Cp, for a specific partition p isproduced according to a procedure as follows. The size of the partitionp is assumed to be M×N pixels (M pixels laterally and N pixelslongitudinally). The partition p is also assumed to belong to a layerLx.

(Step S150) The frequency transform is added to Cp, that is determinedcorresponding to the magnitude relation between M and N.(Step S151) When Cp is empty, the frequency transform is added to Cp,whose transform size is the largest of those of frequency transformswhose transform sizes are smaller than all of the partition sizes.

A detailed procedure of step S150 will be described with reference to aflowchart of FIG. 13.

(Step S160) Using a predetermined value Th3 (that is hereinafter, forexample, Th3=16), the value of Min(M, Th3) is set to be M1 and the valueof Min(N, Th3) is set to be N1. Preferably, the value of Th3 is set tobe the length of one side of the transform size of the frequencytransform whose transform size is the largest square, included in thetransform preset. When a frequency transform having a transform size ofa transform size M1×N1 is present in the transform preset, thisfrequency transform is added to the list of transform candidate, Cp, andthe procedure is advanced to step S161.(Step S161) When M is larger than N (when the partition p is a laterallylong rectangle), the procedure is advanced to step S162 and, in othercases, the procedure is advanced to step S163.(Step S162) When a frequency transform having a transform size of atransform size M1×1 is present in the transform preset, this frequencytransform is added to the list of transform candidate, Cp, and theprocedure comes to an end.(Step S163) When M is smaller than N (when the partition p is alongitudinally long rectangle), the procedure is advanced to step S164and, in other cases, the procedure is advanced to step S165.(Step S164) When a frequency transform having a transform size of atransform size 1×N1 is present in the transform preset, this frequencytransform is added to the list of transform candidate, Cp, and theprocedure comes to an end.(Step S165) The value of M1÷2 is set to be M2 and the value of N1÷2 isset to be N2. When a frequency transform having a transform size of atransform size M2×N2 is present in the transform preset, this frequencytransform is added to the list of transform candidate, Cp, and theprocedure comes to an end. This step is executed when M is equal to N(when the partition p is a square).

The magnitude relation between M and N, and the partition size M×N arepieces of partition shape information.

In the above procedure, when a frequency transform whose transform sizehas a longitudinal (lateral) length shorter than the height (width) ofthe partition is present in the transform preset for a laterally longrectangular (longitudinally long rectangular) partition, this frequencytransform is added to the list of transform candidate, Cp. A frequencytransform whose transform size is a laterally long rectangle(longitudinally long rectangle) is effective for a laterally longrectangular (longitudinally long rectangular) partition as mentionedwith reference to FIG. 6 in the description of the procedure forderiving the prohibited transform list by the transform restrictionderiving portion 104 of the video encoding apparatus 10. Especially, thecases where a border of an object is present in the transform size canbe reduced by using a frequency transform having a transform size whoseshorter side length is extremely short compared to its longer sidelength. Therefore, the concentration effect can be enhanced of theenergy on the low-frequency component of the transform coefficient dueto the frequency transform.

The frequency transform determining portion 112 determines a frequencytransform to be applied to each partition in the expanded MB using thelist of transform candidate that is input thereinto, and outputs thefrequency transform as a transform selection flag. More specifically,the rate distortion cost for a case where each of the frequencytransforms included in the list of transform candidate, Cp, is appliedis calculated, and the frequency transform that minimizes the ratedistortion cost is determined as the frequency transform to be appliedto the partition p.

The variable-length-encoding portion 113 produces the encoded data thatcorresponds to the transform coefficient, the prediction parameter, andthe transform selection flag in the expanded MB, based on thevariable-length encoding, the list of transform candidate, and thetransform selection flag in addition to the transform coefficient andthe prediction parameter that are input thereinto, and outputs theencoded data.

The procedure for variable-length-encoding the transform selection flagfor each partition in the expanded MB is as described at steps S80 toS92 (FIG. 10) by the variable-length-encoding portion 108 of the videoencoding apparatus 10. Steps S133 to S135 by thevariable-length-encoding portion 108 is applied as a detailed procedurefor variable-length-encoding the transform selection flag for a specificpartition.

Operations of the video encoding apparatus 11 will be described.

(Step S170) The input video externally input into the video encodingapparatus 11 is sequentially input in expanded MBs into the predictionparameter determining portion 102 and the prediction residual producingportion 106. Processes of S171 to S179 as follows are sequentiallyexecuted for each of the expanded MBs.(Step S171) The prediction parameter determining portion 102 determinesthe prediction parameter for the expanded MB to be processed based onthe input video that is input thereinto, and outputs the predictionparameter to the predictive image producing portion 103 and thevariable-length-encoding portion 113.(Step S172) The predictive image producing portion 103 produces thepredictive image that approximates an area of the expanded MB to beprocessed in the input video based on the prediction parameter inputthereinto and the local decoded image recorded in the frame memory 101,and outputs the predictive image to the prediction residual producingportion 106 and the local decoded image producing portion 110.(Step S173) The prediction residual producing portion 106 produces theprediction residual that corresponds to the expanded MB to be processed,based on the input video and the predictive image that are inputthereinto, and outputs the prediction residual to the frequencytransform determining portion 112 and the transform coefficientproducing portion 107.(Step S174) The transform candidate deriving portion 111 derives therestriction concerning the frequency transform for each partition in theexpanded MB to be processed, based on the prediction parameter that isinput thereinto, and outputs the restriction to the frequency transformdetermining portion 112 and the variable-length-encoding portion 113.(Step S175) The frequency transform determining portion 112 determinesthe frequency transform to be applied to each partition of the expandedMB to be processed, based on the transform restriction and theprediction residual that are input thereinto, and outputs the frequencytransform as a transform selection flag to the transform coefficientproducing portion 107, the variable-length-encoding portion 113, and theprediction residual reconstructing portion 109.(Step S176) The transform coefficient producing portion 107 applies afrequency transform specified by the transform selection flag inputthereinto to the prediction residual input thereinto, thereby, producesa transform coefficient that corresponds to the expanded MB to beprocessed, and outputs the transform coefficient to thevariable-length-encoding portion 108 and the prediction residualreconstructing portion 109.(Step S177) The prediction residual reconstructing portion 109 appliesan inverse frequency transform that corresponds to the frequencytransform specified by the transform selection flag input thereinto tothe transform coefficient input thereinto, thereby, reconstructs theprediction residual that corresponds to the expanded MB to be processed,and outputs the prediction residual to the local decoded image producingportion 110.(Step S178) The local decoded image producing portion 110 produces thelocal decoded image based on the prediction residual and the predictiveimage that are input thereinto, and outputs the local decoded image tothe frame memory 101 to record the local decoded image in the framememory 101.(Step S179) The variable-length-encoding portion 113variable-length-codes the transform coefficient, the predictionparameter, and the transform selection flag that are input thereinto,using the transform restriction input thereinto, and externally outputsthe encoding results as the encoded data.

According to the above procedure, the video encoding apparatus 11 cancode the input video input thereinto, produce the encoded data, andexternally output the encoded data.

<Another Example of Method of Producing List of Transform Candidate>

An example of the method of producing the list of transform candidate isdescribed in the description concerning the transform candidate derivingportion 111. However, the list of transform candidate may be producedusing another method. For example, when two frequency transforms DCTaand DCTb that are in an analogous relationship (however, the transformsize of DCTa is larger than the transform size of DCTb) are included inthe transform preset, a method of producing the list of transformcandidate is effective which DCTa is added and DCTb is not added to thelist of transform candidate for a partition included in an upper layerand DCTb is added to the list of transform candidate for a partitionincluded in a lower layer. More specifically, when a 16×16 DCT and a 8×8DCT are included in the transform preset, at least the 16×16 DCT isadded and the 8×8 DCT is not added to the list of transform candidatefor a partition included in the layer L0 whose unit of processing is64×64 pixels; and at least the 8×8 DCT is added to the list of transformcandidate for a partition included in the layer L1 whose unit ofprocessing is 32×32 pixels.

Even in the case where the specific frequency transform DCTb (forexample, a 8×8 DCT) cannot be selected for a partition included in thespecific layer Lx, when DCTb can be selected for a partition belongingto the layer Ly that is lower than the layer Lx, an increase of theamount of codes of the encoded data can be suppressed, not by selectingany partition belonging to the upper layer Lx but by selecting thepartition belonging to the lower layer Ly that allows the DCTb to beselected in an area that DCTb is effective. Especially, based on thefact that a frequency transform whose transform size is large iseffective for a large partition, it is effective that, for a partitionbelonging to the upper layer Lx, DCTa having a larger transform size(for example, a 16×16 DCT) is allowed to be selected instead ofprohibiting DCTb from being selected and, on the other hand, for apartition belonging to the lower layer Ly, DCTb is allowed to beselected.

<Configuration of Video Decoding Apparatus 21>

The video decoding apparatus 21 will be described that produces adecoded video by decoding the encoded data encoded by the video encodingapparatus 11.

FIG. 14 is a block diagram of the configuration of the image decodingapparatus 21. The video decoding apparatus 20 includes the frame memory101, the predictive image producing portion 103, the prediction residualreconstructing portion 109, the local decoded image producing portion110, the transform candidate deriving portion 111, and a variable-lengthcode decoding portion 202.

The variable-length code decoding portion 202 decodes the predictionparameter, the transform selection flag, and the transform coefficientbased on the encoded data and the list of transform candidate that areinput thereinto, and outputs the decoding results. More specifically,the variable-length code decoding portion 202 first decodes theprediction parameter from the encoded data and outputs the predictionparameter, then decodes the transform selection flag from the encodeddata using the list of transform candidate and outputs the transformselection flag, and finally decodes the transform coefficient from theencoded data using the transform selection flag and outputs thetransform coefficient. When the transform selection flag is decoded, itis necessary to know how many bits are used to encode the transformselection flag. For that the information of the elements included in thelist of transform candidate is not necessarily needed but it is neededto only know the number of elements included in the list of transformcandidate. In this case, a signal input into the variable-length codedecoding portion 202 and a signal used to decode the transform selectionflag may be only a signal concerning the number of elements included inthe list of transform candidate of the list of transform candidate.

<Operations of Video Decoding Apparatus 21>

Operations of the video decoding apparatus 21 will be described.

(Step S180) The encoded data externally input into the video decodingapparatus 20 is sequentially input into the variable-length codedecoding portion 201 expanded MB by expanded MB, and followingprocessing of steps S181 to S187 is sequentially executed for theencoded data that corresponds to each expanded MB.(Step S181) The variable-length code decoding portion 202 decodes theprediction parameter that corresponds to the expanded MB to be processedfrom the encoded data input thereinto, and outputs the predictionparameter to the predictive image producing portion 103 and thetransform candidate deriving portion 111.(Step S182) The transform candidate deriving portion 111 derives thelist of transform candidate for each partition of the expanded MB to beprocessed based on the prediction parameter input thereinto, and outputsthe list of transform candidate to the variable-length code decodingportion 202.(Step S183) The variable-length code decoding portion 202 decodes thetransform selection flag that corresponds to the MB to be processedbased on the encoded data and the transform restriction that are inputthereinto, and outputs the transform selection flag to the predictionresidual reconstructing portion 109.(Step S184) The variable-length code decoding portion 202 decodes thetransform coefficient that corresponds to the expanded MB to beprocessed based on the encoded data input thereinto and the transformselection flag derived at (step S183), and outputs the transformcoefficient to the prediction residual reconstructing portion 109.(Step S185) The predictive image producing portion 103 produces thepredictive image that corresponds to the expanded MB to be processedbased on the prediction parameter input thereinto and the local decodedimage recorded in the frame memory 101, and outputs the predictive imageto the local decoded image producing portion 110.(Step S186) The prediction residual reconstructing portion 109 appliesthe inverse frequency transform that corresponds to the frequencytransform specified by the transform selection flag input thereinto, tothe transform coefficient input thereinto, thereby, reconstructs theprediction residual that corresponds to the expanded MB to be processed,and outputs the prediction residual to the local decoded image producingportion 110.(Step S187) The local decoded image producing portion 110 produces thelocal decoded image based on the prediction residual and the predictiveimage that are input thereinto, outputs the local decoded image to theframe memory 101 to record the local decoded image in the frame memory101, and externally outputs the local decoded image as the area on thedecoded video that corresponds to the block to be processed.

<Conclusion for Decoder>

As above, according to the video decoding apparatus 21, the decodedvideo can be produced from the encoded data that is produced by thevideo encoding apparatus 11.

Third Embodiment

A video encoding apparatus 30 and a video decoding apparatus 40 that areyet another embodiment of the video encoding apparatus and the videodecoding apparatus according to the present invention will be describedwith reference to FIGS. 15 to 16. In the description of the accompanyingdrawings, the same reference numerals are given to the same componentsand those components will not again be described. It is assumed that thepartition structure and the transform preset that are available for thevideo encoding apparatus 30 and the video decoding apparatus 40 are thesame as those that are used for the video encoding apparatus 11 and thevideo decoding apparatus 21.

The video encoding apparatus 30 and the video decoding apparatus 40 inthe embodiment are different from the video encoding apparatus 11 andthe video decoding apparatus 21 in that the video encoding apparatus 30and the video decoding apparatus 40 include a function that adaptivelychanges the method of deriving the list of transform candidate by thetransform candidate deriving portion matching the property of the videoin a predetermined unit that is larger than an MB such as a scene, aframe, or a slice of the video.

FIG. 15 is a block diagram of the configuration of the video encodingapparatus 30. The video encoding apparatus 30 includes the frame memory101, the prediction parameter determining portion 102, the predictiveimage producing portion 103, the prediction residual producing portion106, the transform coefficient producing portion 107, the predictionresidual reconstructing portion 109, the local decoded image producingportion 110, the frequency transform determining portion 112, a list oftransform candidate deriving rule determining portion 301, a transformcandidate deriving portion 302, and a variable-length-encoding portion303.

The list of transform candidate deriving rule determining portion 301produces a list of transform candidate deriving rule that specifies orupdates the method of deriving the list of transform candidate executedby the transform candidate deriving portion based on the input videothat is input in predetermined units that are larger than an MB such asa scene, a frame, or a slice. Hereinafter, for the simplicity of thedescription, the list of transform candidate deriving rule will bedescribed assuming that this rule is produced for each frame.

(Definition of List of Transform Candidate Deriving Rule)

The list of transform candidate deriving rule is defined as acombination of the basic rules that are listed as follows.

Basic Rule 1: This rule specifies to add a predetermined frequencytransform B in the transform preset to the list of transform candidatefor a predetermined partition A. Hereinafter, the basic rule 1 isdescribed in a form of [permission, partition A, frequency transform B].For example, [permission, 64×64, T16×16] indicates to add a frequencytransform of T16×16 to the list of transform candidate for a64×64-partition.

Basic Rule 2: This rule specifies to prohibit the predeterminedfrequency transform B in the transform preset from being included in thelist of transform candidate for the predetermined partition A.Hereinafter, the basic rule 2 is described in a form of [prohibition,partition A, frequency transform B]. For example, [prohibition, 64×64,T4×4] indicates to prohibit T4×4 for a partition haying a size of 64×64and not to include T4×4 in the list of transform candidate.

Basic Rule 3: This rule specifies to replace the frequency transform Bin the list of transform candidate with another frequency transform Cwhen the predetermined frequency transform B in the transform preset isincluded in the list of transform candidate for the predeterminedpartition A. Hereinafter, the basic rule 3 is described in a form of[replacement, partition A, frequency transform B, frequency transformC]. For example, [replacement, 64×32, T4×4, T16×1] indicates to exclude,T4×4 from the list of transform candidate and, instead, to add T16×1 tothe list of transform candidate when T4×4 is included in the list oftransform candidate for a partition having a size of 64×32.

The list of transform candidate deriving rule includes a plurality ofbasic rules and each of the basic rules is classified into any one ofthe basic rules 1 to 3.

In the list of transform candidate deriving rule, in addition to thebasic rules, or instead of the basic rules, a complex rule expressed bya combination of the basic rules may be included. Examples of some ofthe complex rules will be listed.

Complex Rule 1: A specific transform is prohibited for a partition thatbelongs to a specific layer. For example, a rule of prohibiting anytransform whose size is T8×8 or smaller in the L0 layer corresponds tothis complex rule 1. The complex rule (R1) can be expressed as a set ofthe basic rules as follows.

R1={[prohibition, P,T]:(“P” is a partition that belongs to the L0layer)Λ(“T” is a frequency transform of T8×8 or smaller)}

A rule also corresponds to this complex rule 1, of prohibiting anyfrequency transform whose size is small in the frequency transforms inan analogous relationship in a layer that is upper than a predeterminedlayer, more specifically, a rule of prohibiting T8×8 and T4×4 of thetransforms of T16×16, T8×8, and T4×4 that are in an analogousrelationship in a layer that is upper than the layer L1.

Complex Rule 2: A specific transform A is replaced with a specifictransform B for a partition having a specific shape. For example, a ruleof replacing a rectangular frequency transform with a specific squarefrequency transform (for example, T4×4) for a square partitioncorresponds to this complex rule 2. The complex rule (R2) can beexpressed as a set of the basic rules as follows:

R2={[replacement, P,T,T4×4]:(Pεsquare

partitions)Λ(Tεrectangular frequency transforms)}

A rule of replacing a square frequency transform with a laterally longrectangular frequency transform for a laterally long rectangularpartition also corresponds to this complex rule 2.

(Procedure for Determining List of Transform Candidate Deriving Rule)

Candidate rules having, as its elements, the basic rules and the complexrules are specified in advance before the start of the encodingprocessing, and the list of transform candidate deriving rule is set tobe empty. The rate distortion cost is calculated for a case where theencoding processing is executed applying each of the basic rules or eachof the complex rules included in the candidate rules to each frame thatis input. A rate distortion cost C1 is also calculated for a case whereall the candidate rules are not applied. In comparison between a ratedistortion cost C2 calculated for a case where each of the basic rulesor each of the complex rules is applied, and the cost C1, when the costC2 is smaller than the cost C1, it is determined that the basic rule orthe complex rule is applied and included in the list of transformcandidate deriving rule.

According to the above procedure, only the basic rule or the complexrule of predetermined candidate rules which can reduce the ratedistortion cost by applying when encoding a frame, is added to the listof transform candidate deriving rule.

The transform candidate deriving portion 302 outputs, the information onfrequency transforms that can be selected in each partition in theexpanded MB as a list of transform candidate based on the predictionparameter that is input and the list of transform candidate derivingrule. The list of transform candidate is correlated with each partitionin the expanded MB and specifies a set of the frequency transforms thatcan be selected in each partition of the frequency transforms includedin the transform preset. At this step, the list of transform candidatederiving rule that is input is also used for the processing of derivingthe list of transform candidate.

A procedure is as follows for producing the list of transform candidate,Cp, for the specific partition p based on the list of transformcandidate deriving rule that is input. It is assumed that the size ofthe partition p is M×N pixels (laterally, M pixels and, longitudinally,N pixels).

(Step S200) When the complex rules are included in the list of transformcandidate deriving rule, each of the complex rules is disassembled intothe basic rules and these basic rules are added to the list of transformcandidate deriving rule.(Step S201) Processing of step S202 is executed for all the basic rulesbelonging to the basic rule 1 that are included in the list of transformcandidate.(Step S202) The basic rule 1 to be processed is expressed as[permission, P1, T1]. When the shape of the partition p and P1 coincidewith each other, a frequency transform T1 is added to the list oftransform candidate.(Step S203) Processing of step S204 is executed for all the basic rulesbelonging to the basic rule 2 that are included in the list of transformcandidate.(Step S204) The basic rule 2 to be processed is expressed as[prohibition, P2, T2]. When the shape of the partition p and P2 coincidewith each other and a frequency transform T2 is present in the list oftransform candidate, the frequency transform T2 is removed from the listof transform candidate.(Step S205) Processing of step S206 is executed for all the basic rulesbelonging to the basic rule 3 that are included in the list of transformcandidate.(Step S206) The basic rule 2 to be processed is expressed as[replacement, P3, T3, T4]. When the shape of the partition p and P3coincide with each other and a frequency transform T3 is present in thelist of transform candidate, the frequency transform T3 is replaced witha frequency transform T4.

According to the above procedure, the transform candidate derivingportion 302 can derive the list of transform candidate according to thelist of transform candidate deriving rule input thereinto.

The variable-length-encoding portion 303 produces encoded data thatcorresponds respectively to the transform coefficient, the predictionparameter, the list of transform candidate, the transform selectionflag, and the list of transform candidate deriving rule that are inputthereinto, and outputs the encoded data.

Details of the processing of producing the encoded data corresponding tothe list of transform candidate deriving rule will be described. Theencoded data is produced by variable-length-encoding each of the basicrules or the complex rules that are included in the list of transformcandidate deriving rule. In the variable-length-encoding of the basicrule, the information is encoded first that indicates which of the basicrules 1 to 3 the basic rule to be encoded is classified into, and theninformation is encoded that indicates the partition to which the basicrule is applied. Finally, information that indicates the permittedfrequency transform is encoded in the case of the basic rule 1,information that indicates the prohibited frequency transform is encodedin the case of the basic rule 2, and information that indicates the kindof each frequency transform before and after the replacement is encodedin the case of the basic rule 3. When which basic rule can be includedin the list of transform candidate deriving list is determined inadvance, the amount of codes can be reduced by determining theinformation indicating whether the basic rule is applied to be theencoded data instead of variable-length-encoding the basic rulesaccording to the above method. When it is determined in advance that aspecific basic rule is always applied it is not necessary tovariable-length-code the basic rule.

A complex rule is encoded after the complex rule is disassembled intothe basic rules. When which complex rule can be included in the list oftransform candidate deriving list is determined in advance, the amountof codes can be reduced by determining the information indicatingwhether the complex rule is applied to be the encoded data. For example,it is possible to encode where apply or not a complex rule thatprohibits T4×4 and T8×8 in a partition that is larger than 32×32 as aflag of one bit.

It may be possible to encode the information indicating with or withoutapplication for each of the basic rules included in a rule group whichis specified by collectively handling the specific basic rules or thespecific complex rules, to encode a flag indicating where or notestimating with or without application for all the basic rules includedin the rule group. More specifically, when complex rules indicatingwhether T16×16, T8×8, and T4×4 are applied in the layer L3 arerespectively expressed as “enable_t16×16_L3”, “enable_t16×16_L3” and“enable_t16×16_L3”, a rule group “enable_L3” is produced by collectivelyhandling these three complex rules. In the encoding, whether enable_L3is applied is encoded by one bit first. When enable_L3 is applied,whether each complex rule included in the rule group is applied isencoded by one bit. When enable_L3 is not applied, whether each complexrule is applied is estimated according to a predetermined method.

The basic rules and the complex rules may collectively be encodedwithout variable-length-encoding those rules one by one. For example,only when a flag is encoded that indicates whether all the basic rulesare not applied or at least one basic rule is applied and this flagindicates that at least one basic rule is applied, the information maybe encoded that indicates whether each of the basic rules is applied. Inaddition, a flag may be encoded that indicates whether the list oftransform candidate deriving rule applied in a previous frame iscontinuously applied and, only when this list of transform candidatederiving rule is not continuously applied, the list of transformcandidate deriving rule may be encoded.

Operations of the video encoding apparatus 30 will be described.

(Step S210) The input video externally input into the video encodingapparatus 30 is input into the list of transform candidate deriving ruledetermining portion 301 frame by frame, and is sequentially input intothe prediction parameter determining portion 102 and the predictionresidual producing portion 106 expanded MB by expanded MB. Processes ofsteps S211 to S212 are executed for each frame and processes of stepsS213 to S221 are executed for each expanded MB.(Step S211) The list of transform candidate deriving rule determiningportion 301 produces the list of transform candidate deriving rule basedon the frame input thereinto, and outputs the list of transformcandidate deriving rule to the transform candidate deriving portion 302and the variable-length-encoding portion 303.(Step S212) The variable-length-encoding portion 303 produces thecorresponding encoded data based on the list of transform candidatederiving rule input thereinto, and externally outputs the correspondingencoded data.(Step S213) The prediction parameter determining portion 102 determinesthe prediction parameter for the expanded MB to be processed based onthe input video input thereinto, and outputs the prediction parameter tothe predictive image producing portion 103, the transform candidatederiving portion 302, and the variable-length-encoding portion 303.(Step S214) The predictive image producing portion 103 produces thepredictive image that approximates an area of the expanded MB to beprocessed in the input video, based on the prediction parameter inputthereinto and the local decoded image recorded in the frame memory 101,and outputs the predictive image to the prediction residual producingportion 106 and the local decoded image producing portion 110.(Step S215) The prediction residual producing portion 106 produces theprediction residual that corresponds to the expanded MB to be processedbased on the input video and the predictive image that are inputthereinto, and outputs the prediction residual to the frequencytransform determining portion 112 and the transform coefficientproducing portion 107.(Step S216) The transform candidate deriving portion 302 derives therestriction concerning the frequency transform for each partition of theexpanded MB to be processed based on the prediction parameter and thelist of transform candidate deriving rule that are input thereinto, andoutputs the restriction to the frequency transform determining portion112 and the variable-length-encoding portion 303.(Step S217) The frequency transform determining portion 112 determinesthe frequency transform to be applied to each partition of the expandedMB to be processed based on the transform restriction and the predictionresidual that are input thereinto, and outputs the frequency transformas the transform selection flag to the transform coefficient producingportion 107, the variable-length-encoding portion 303, and theprediction residual reconstructing portion 109.(Step S218) The transform coefficient producing portion 107 applies thefrequency transform specified by the transform selection flag inputthereinto to the prediction residual input thereinto, thereby, producesthe transform coefficient that corresponds to the expanded MB to beprocessed, and outputs the transform coefficient to thevariable-length-encoding portion 108 and the prediction residualreconstructing portion 109.(Step S219) The prediction residual reconstructing portion 109 appliesthe inverse frequency transform that corresponds to the frequencytransform specified by the transform selection flag input thereinto, tothe transform coefficient input thereinto, thereby, reconstructs theprediction residual that corresponds to the expanded MB to be processed,and outputs the prediction residual to the local decoded image producingportion 110.(Step S220) The local decoded image producing portion 110 produces thelocal decoded image based on the prediction residual and the predictiveimage that are input thereinto, and outputs the local decoded image tothe frame memory 101 to record the local decoded image in the framememory 101.(Step S221) The variable-length-encoding portion 303variable-length-codes the transform coefficient, the predictionparameter, and the transform selection flag that are input thereinto,using the transform restriction input thereinto, and externally outputsthe resultant data as the encoded data.

According to the above procedure, the video encoding apparatus 30 cancode the input video that is input thereinto, thereby, produce theencoded data, and externally output the encoded data.

<Configuration of Video Decoding Apparatus 40>

The video decoding apparatus 40 will be described that decodes theencoded data that is encoded by the video encoding apparatus 30 andthat, thereby, produces the decoded video.

FIG. 16 is a block diagram of the configuration of the image decodingapparatus 40. The video decoding apparatus 40 includes the frame memory101, the predictive image producing portion 103, the prediction residualreconstructing portion 109, the local decoded image producing portion110, the transform candidate deriving portion 302, and a variable-lengthcode decoding portion 401.

The variable-length code decoding portion 401 decodes the predictionparameter, the transform selection flag, the transform coefficient, andthe list of transform candidate deriving rule based on the encoded dataand the list of transform candidate that are input thereinto, andoutputs the decoding results. More specifically, the variable-lengthcode decoding portion 401: first decodes and outputs the list oftransform candidate deriving rule; then, decodes the predictionparameter from the encoded data and outputs the prediction parameter;decodes the transform selection flag from the encoded data using thelist of transform candidate and outputs the transform selection flag;and, finally, decodes the transform coefficient from the encoded datausing the transform selection flag and outputs the transformcoefficient.

<Operations of Video Decoding Apparatus 40>

Operations of the video decoding apparatus 40 will be described.

(Step S230) The encoded data externally input into the video decodingapparatus 40 is sequentially input into the variable-length codedecoding portion 401 in frames. Processes of S231 to S239 as follows aresequentially executed for the encoded data that corresponds to eachframe.(Step S231) The variable-length code decoding portion 401 decodes thelist of transform candidate deriving rule that corresponds to the frameto be processed, from the encoded data input thereinto, and outputs thelist of transform candidate deriving rule to the transform candidatederiving portion 302.(Step S232) The variable-length code decoding portion 401 divides theencoded data for each frame input thereinto into encoded data for eachexpanded MB. Processes of steps S233 to S239 as follows are sequentiallyexecuted for encoded data corresponding to each expanded MB.(Step S233) The variable-length code decoding portion 401 decodes theprediction parameter from each of encoded data corresponding to anexpanded MB to be processed, and outputs the prediction parameter to thetransform candidate deriving portion 302.(Step S234) The transform candidate deriving portion 302 derives thelist of transform candidate for each partition of the expanded MB to beprocessed based on the list of transform candidate deriving rule and theprediction parameter that are input thereinto, and outputs the list oftransform candidate to the variable-length code decoding portion 401.(Step S235) The variable-length code decoding portion 401 decodes thetransform selection flag that corresponds to the MB to be processedbased on the encoded data and the transform restriction that are inputthereinto, and outputs the transform selection flag to the predictionresidual reconstructing portion 109.(Step S236) The variable-length code decoding portion 202 decodes thetransform coefficient that corresponds to the expanded MB to beprocessed, based on the encoded data input thereinto and the transformselection flag derived at (step S235), and outputs the transformcoefficient to the prediction residual reconstructing portion 109.(Step S237) The predictive image producing portion 103 produces thepredictive image that corresponds to the expanded MB to be processedbased on the prediction parameter input thereinto and the local decodedimage recorded in the frame memory 101, and outputs the predictive imageto the local decoded image producing portion 110.(Step S238) The prediction residual reconstructing portion 109 appliesthe inverse frequency transform that corresponds to the frequencytransform specified by the transform selection flag input thereinto, tothe transform coefficient input thereinto, thereby, reconstructs theprediction residual that corresponds to the expanded MB to be processed,and outputs the reconstructed prediction residual to the local decodedimage producing portion 110.(Step S239) The local decoded image producing portion 110 produces thelocal decoded image based on the prediction residual and the predictiveimage that are input thereinto, outputs the local decoded image to theframe memory 101 to record the local decoded image in the frame memory101, and externally outputs the local decoded image as the area on thedecoded video that corresponds to the block to be processed.

As above, according to the video decoding apparatus 40, the decodedvideo can be produced from the encoded data produced by the videoencoding apparatus 11.

A portion or all of the video encoding apparatus and the video decodingapparatus in each of the embodiments may typically be implemented as anLSI (Large Scale Integration) that is an integrated circuit. Each of thefunctional blocks of the video encoding apparatus and the video decodingapparatus may individually be implemented as a chip, or a portion or allof those functional blocks may be integrated into one chip. The approachof implementation thereof as an integrated circuit(s) may be realized bynot only an LSI but also a dedicated circuit(s) or a multi-purposeprocessor(s). When a technique of implementation thereof as integratedcircuit(s) is established that supersedes LSIs due to the advancement ofthe semiconductor technology, such technique may also be used.

EXPLANATIONS OF LETTERS OR NUMERALS

10 . . . video encoding apparatus, 11 . . . video encoding apparatus, 20. . . video decoding apparatus, 21 . . . video decoding apparatus, 30 .. . video encoding apparatus, 40 . . . video decoding apparatus, 101 . .. frame memory, 102 . . . prediction parameter determining portion, 103. . . predictive image producing portion, 104 . . . transformrestriction deriving portion, 105 . . . frequency transform determiningportion, 106 . . . prediction residual producing portion, 107 . . .transform coefficient producing portion, 108 . . .variable-length-encoding portion, 109 . . . prediction residualreconstructing portion, 110 . . . local decoded image producing portion,111 . . . transform candidate deriving portion, 112 . . . frequencytransform determining portion, 113 . . . variable-length-encodingportion, 201 . . . variable-length code decoding portion, 202 . . .variable-length code decoding portion, 301 . . . candidate list derivingrule determining portion, 302 . . . transform candidate derivingportion, 303 . . . variable-length-encoding portion, 401 . . .variable-length code decoding portion.

1-19. (canceled)
 20. A video encoding apparatus that divides an inputvideo into predetermined size blocks and executes an encoding processingfor each block, comprising: a prediction parameter determining portionthat determines a partition structure of the block; a predictive imageproducing portion that produces a predictive image for each partitionspecified by the partition structure; a transform coefficient producingportion that applies any one of transforms included in a predeterminedtransform preset to a prediction residual that is a difference betweenthe predictive image and the input video; a transform candidate derivingportion that determines a list of transform candidate that is a list ofapplicable transforms based on partition shape information; a frequencytransform determining portion that, for each of the blocks, determines atransform selection flag indicating transforms to be applied to theprediction residual in the block from among transforms included in thelist of transform candidate; and a variable-length-encoding portion thatvariable-length-codes the transform selection flag based on the list oftransform candidate.
 21. The video encoding apparatus as defined inclaim 20, further comprising a transform restriction deriving portionthat produces a prohibited transform list that is a list of transformsinapplicable to each partition based on the partition shape information,wherein the variable-length-encoding portion variable-length-encodes thetransform selection flag based on the list of transform candidate thatis derived based on the prohibited transform list and the transformpreset.
 22. The video encoding apparatus as defined in claim 20, whereinthe partition shape information is a ratio of a longitudinal length to alateral length of a partition, or a magnitude relation between thelongitudinal length and the lateral length of the partition.
 23. Thevideo encoding apparatus as defined in claim 20, wherein the partitionstructure is expressed by a layer structure, and specifies that eachpartition is included in either layer corresponding to a shape of thepartition, and the partition shape information includes a layer that thepartition belongs to.
 24. The video encoding apparatus as defined inclaim 20, wherein the predetermined transform preset includes at leastone transform whose transform size is a square and at least onetransform whose transform size is a laterally long rectangle or alongitudinally long rectangle, when a lateral length of a partition islonger than a longitudinal length thereof, the transform candidatederiving portion includes at least one laterally long rectangletransform in the list of transform candidate, when a longitudinal lengthof a partition is longer than a lateral length thereof, the transformcandidate deriving portion includes at least one longitudinally longrectangle transform in the list of transform candidate, and when alongitudinal length of a partition is equal to a lateral length thereof,the transform candidate deriving portion includes at least one squaretransform in the list of transform candidate.
 25. The video encodingapparatus as defined in claim 20, wherein the predetermined transformpreset includes at least one or more transforms whose transform size isa laterally long rectangle whose height is one pixel, and when a laterallength of a partition is longer than a longitudinal length thereof, thetransform candidate deriving portion includes in the list of transformcandidate a transform whose transform size is a laterally long rectanglewhose height is one pixel.
 26. The video encoding apparatus as definedin claim 21, wherein the predetermined transform preset includes atleast two or more transforms whose transform sizes mutually are in ananalogous relationship, and when each of the smallest values of alongitudinal length and a lateral length of a partition is equal to orlarger than a predetermined threshold value, the transform restrictionderiving portion includes in the prohibited transform list a transformwhose transform size is the smallest among those of transforms havingthe transform size bearing an analogous relationship with each other.27. The video encoding apparatus as defined in claim 23, wherein thepredetermined transform preset includes a first transform and a secondtransform having an equal magnitude relation between a longitudinallength and a lateral length in the transform size to that of the firsttransform and having smaller transform size than that of the firsttransform, the partition structure is expressed by a layer structure andspecified that each partition is included in either layer correspondingto a shape of the partition, and the transform candidate derivingportion includes the first transform in the list of transform candidateand does not include the second transform in the list of transformcandidate when a partition belongs to a predetermined layer that is nota lowermost layer, and includes the second transform in the list oftransform candidate when the partition belongs to a layer lower than thepredetermined layer that is not the lowermost layer.
 28. A videodecoding apparatus that executes a decoding processing for input encodeddata for each block, comprising: a variable-length decoding portion thatdecodes a partition structure of a block to be processed from the inputencoded data; a predictive image producing portion that produces apredictive image for each partition that is specified by the partitionstructure; and a transform candidate deriving portion that determines alist of transform candidate that is a list of applicable transformsbased on partition shape information, wherein thevariable-length-decoding portion decodes a transform selection flagbased on the input decoded data and the list of transform candidate aswell as decodes a transform coefficient of the block to be processedbased on the transform selection flag, the video decoding apparatusfurther comprises: a prediction residual reconstructing portion thatreconstructs a prediction residual by applying inverse transforms to thetransform coefficient, the inverse transforms corresponding totransforms, the transforms being specified by the transform selectionflag; and a local decoded image producing portion that outputs decodedimage data based on the predictive image and the prediction residual,the decoded image data corresponding to the block to be processed. 29.The video decoding apparatus as defined in claim 28, further comprisinga transform restriction deriving portion that produces a prohibitedtransform list that is a list of transforms inapplicable to eachpartition based on the partition shape information, wherein thevariable-length-decoding portion variable-length-decodes the transformselection flag based on the list of transform candidate that is derivedbased on the prohibited transform list and the transform preset.
 30. Thevideo decoding apparatus as defined in claim 28, wherein the partitionshape information is a ratio of a longitudinal length to a laterallength of a partition, or a magnitude relation between the longitudinallength and the lateral length of the partition.
 31. The video decodingapparatus as defined in claim 28, wherein the partition structure isexpressed by a layer structure, and specifies that each partition isincluded in either layer, and the partition shape information includes alayer that the partition belongs to.
 32. The video decoding apparatus asdefined in claim 28, wherein the predetermined transform preset includesat least one transform whose transform size is a square and at least onetransform whose transform size is a laterally long rectangle or alongitudinally long rectangle, when a lateral length of a partition islonger than a longitudinal length thereof, the transform candidatederiving portion derives a list of transform candidate including atleast one laterally long rectangle transform, when a longitudinal lengthof a partition is longer than a lateral length thereof, the transformcandidate deriving portion derives a list of transform candidateincluding at least one longitudinally long rectangle transform, and whena longitudinal length of a partition is equal to a lateral lengththereof, the transform candidate deriving portion derives a list oftransform candidate including at least one square transform.
 33. Thevideo decoding apparatus as defined in claim 29, wherein thepredetermined transform preset includes at least two or more transformswhose transform sizes mutually are in an analogous relationship, andwhen each of the smallest values of a longitudinal length and a laterallength of a partition is equal to or larger than a predeterminedthreshold value, the transform candidate deriving portion derives a listof transform candidate excluding a transform whose transform size is thesmallest among those of transforms bearing an analogous relationshipwith each other.
 34. The video decoding apparatus as defined in claim31, wherein the predetermined transform preset includes a firsttransform and a second transform having an equal magnitude relationbetween a longitudinal length and a lateral length in the transform sizeto that of the first transform and having smaller transform size thanthat of the first transform, the partition structure is expressed by alayer structure, and specifies that each partition is included in eitherlayer corresponding to a shape of the partition, and the transformcandidate deriving portion includes the first transform and excludes thesecond transform from the list of transform candidate when a partitionbelongs to a predetermined layer that is not a lowermost layer, andincludes the second transform in the list of transform candidate whenthe partition belongs to a layer lower than the predetermined layer thatis not the lowermost layer.